Reflective video display apparatus for interactive training and demonstration and methods of using same

ABSTRACT

A method includes capturing, using an Internet of Things (IoT) device, real-time video of a user performing an exercise associated with a first exercise class. Real-time video of the user is displayed, via a display, concurrently with video of an instructor, to provide a visual comparison of the user to the instructor. Image analysis of the real-time video of the user is performed to determine a performance of the user, and a representation thereof is displayed. Biometric data associated with the user is received at the IoT device from a wearable device at multiple points in time. Heart rates are identified based on the biometric data, and scores based on the heart rates are displayed via the display. A recommendation for a second exercise class different from the first exercise class is determined based on a profile of the user, and displayed via the display.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/188,862, filed Mar. 1, 2021 and titled “Reflective Video DisplayApparatus for Interactive Training and Demonstration and Methods ofUsing Same,” which is a continuation of U.S. patent application Ser. No.17/062,069, filed Oct. 2, 2020 and titled “Reflective Video DisplayApparatus for Interactive Training and Demonstration and Methods ofUsing Same,” now U.S. Pat. No. 10,981,047, which is a continuation ofU.S. patent application Ser. No. 16/672,103, filed on Nov. 1, 2019 andtitled “Reflective Video Display Apparatus for Interactive Training andDemonstration and Methods of Using Same,” now U.S. Pat. No. 10,828,551,which is a bypass Continuation of International Patent Application No.PCT/US2019/034292, filed on May 29, 2019 and titled “A Reflective VideoDisplay Apparatus for Interactive Training and Demonstration and Methodsof Using Same,” which claims priority to and the benefit of U.S.Provisional Patent Application No. 62/677,351, filed on May 29, 2018 andtitled “System for Displaying Two-Way Interactive Data for Training andDemonstration on a Reflective Video Display and Scheme for Serial DataTransfer Over Bluetooth Low Energy,” the contents of each of which areincorporated herein by reference in their entireties.

BACKGROUND

Exercise is an important part of maintaining an individual's health andwellbeing. For many people, exercising is an activity that typicallyinvolves going to a gymnasium where they partake in a workout guided byan instructor (e.g., a fitness instructor, a personal trainer). However,dedicating a regular period of time to exercise at a gym can be achallenging endeavor due to other commitments in one's daily life (e.g.,a person's job, family obligations). Oftentimes, a gym may be located atan inconvenient location and/or an instructor's availability is limitedto certain periods of time during the day, thus limiting a person'sability to attend a workout at the gym. This inconvenience may also bedetrimental to the instructor whose clientele may be restricted topeople who are able to attend their workout at the gym at the prescribedperiod of time.

SUMMARY

Embodiments described herein are directed to a reflective video displayapparatus (also referred to herein as a “smart mirror” and an“interactive exercise system”) configured to display video content, suchas a pre-recorded or live workout led by an instructor, to a user andprovide an interface that allows the user to interact and personalizethe video content. The smart mirror may be a networked devicecommunicatively coupled to a content provider (e.g., a server, a cloudservice) and/or a smart device (e.g., a smart phone, a tablet, acomputer). The smart mirror may include a display panel and a speaker tooutput video content and audio to the user. The smart mirror may alsoinclude a camera and a microphone to capture video and audio of the userduring a workout. The smart mirror may thus enable two-way communicationbetween the user and the instructor during a workout. In this manner,the smart mirror may provide a convenient option for the user to receivea guided workout while enabling greater personalization and individualguidance similar to a workout provided by a personal trainer orinstructor at a conventional gymnasium.

One example of a smart mirror includes a communications interface toreceive video imagery of an exercise instructor, a display, operablycoupled to the communications interface, to show the video imagery ofthe exercise instructor, and a mirror, disposed in front of the display,to reflect an image of a person opposite the display. The mirror has apartially reflecting section to transmit the video imagery of theexercise instructor to the person opposite the display such that thevideo imagery of the exercise instructor appears superimposed on aportion of the image of the person.

Another example of a smart mirror includes a mirror having a partiallyreflecting section and a fully reflecting section, a display arranged todisplay exercise content through the partially reflecting section of themirror where the exercise content includes (1) video imagery of anexercise instructor, (2) biometric data of a first person exercisingwhile viewing the exercise content, and (3) information about a secondperson watching the video imagery on another interactive exercisesystem, and a frame, disposed behind the mirror, to hold the displayopposite the partially reflecting section of the mirror where the framehas a width equal to or smaller than a width of the mirror and a heightequal to or smaller than a height of the mirror.

One example of an interactive exercise method includes the follow steps:(1) streaming exercise content to an interactive video system comprisinga mirror having a partially reflecting section and a display disposed onone side of the partially reflecting section, (2) displaying theexercise content to a user via the display and the partially reflectingsection of the mirror, and (3) reflecting an image of the user with themirror such that the image of the user appears at least partiallysuperimposed on the exercise content displayed via the display and thepartially reflecting section of the mirror.

One example of a method of using a smart mirror includes the followingsteps while displaying exercise content to a user on a video displaybehind a partially transmissive mirror: (1) reflecting an image of theuser with the partially transmissive mirror, (2) measuring a heart rateof the user with a heart rate monitor attached to the user, (3)transmitting the heart rate from the heart rate monitor to an antennaoperably coupled to the video display, (4) displaying the heart rate ofthe user on the video display, and (5) displaying a target heart ratefor the user on the video display.

One example of a method of exercising using the smart mirror includesthe following steps: (1) displaying an exercise video on the smartmirror to a person, (2) displaying a first target heart rate zone forthe person to reach during a first segment of the exercise video wherethe first target heart rate zone is based on an exercise displayedduring the first segment of the exercise video and at least one of theperson's age, height, weight, exercise history, or preference, (3)displaying a second target heart rate zone for the person to reachduring a second segment of the exercise video where the second targetheart rate zone is different from the first target heart rate zone andbased on an exercise displayed during the second segment of the exercisevideo and the at least one of the person's age, gender, height, weight,exercise history, or preference, (4) acquiring heart rate data from theheart rate monitor while displaying the first segment of the exercisevideo and the second segment of the exercise video on the smart mirrorand while transitioning from the first segment of the exercise video tothe second segment of the exercise video, (5) determining a score basedon a change in the heart rate data associated with a transition from thefirst segment of the exercise video to the second segment of theexercise video, and (6) displaying an indication of the score on thesmart mirror while displaying the exercise video to the person.

All combinations of the foregoing concepts and additional conceptsdiscussed in greater detail below (provided such concepts are notmutually inconsistent) are contemplated as being part of the inventivesubject matter disclosed herein. In particular, all combinations ofclaimed subject matter appearing at the end of this disclosure arecontemplated as being part of the inventive subject matter disclosedherein. Terminology explicitly employed herein that also may appear inany disclosure incorporated by reference should be accorded a meaningmost consistent with the particular concepts disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The skilled artisan will understand that the drawings primarily are forillustrative purposes and are not intended to limit the scope of theinventive subject matter described herein. The drawings are notnecessarily to scale; in some instances, various aspects of theinventive subject matter disclosed herein may be shown exaggerated orenlarged in the drawings to facilitate an understanding of differentfeatures. In the drawings, like reference characters generally refer tolike features (e.g., functionally similar and/or structurally similarelements).

FIG. 1 shows a block diagram of an exemplary smart mirror.

FIG. 2A shows an exemplary smart mirror with a stand disposed on thebottom.

FIG. 2B shows another view of a smart mirror with a stand disposed onthe bottom.

FIG. 2C shows an exemplary smart mirror mounted to a wall.

FIG. 3A shows a front, perspective view of an exemplary smart mirror.

FIG. 3B shows an exploded view of the smart mirror of FIG. 3A.

FIG. 3C shows an exploded view of an upper electronics assembly in thesmart mirror of FIG. 3A.

FIG. 3D shows an exploded view of a lower electronics assembly in thesmart mirror of FIG. 3A.

FIG. 4A shows a front, perspective view of an exemplary an inner frame.

FIG. 4B shows a left-side view of the inner frame of FIG. 4A.

FIG. 4C shows a front-side view of the inner frame of FIG. 4A.

FIG. 4D shows a front-side, flat representation of the inner frame ofFIG. 4A.

FIG. 4E shows a front-side, flat representation of the inner frame ofFIG. 4A with various labeled holes for assembly.

FIG. 5A shows a front, perspective view of an exemplary an outer shell.

FIG. 5B shows a front-side view of the outer shell of FIG. 5A.

FIG. 5C shows a side-side view of the outer shell of FIG. 5A.

FIG. 5D shows a top-side view of the outer shell of FIG. 5A.

FIG. 5E shows a bottom-side view of the outer shell of FIG. 5A.

FIG. 5F a front-side, flat representation of the outer shell of FIG. 5Awith various labeled holes for assembly.

FIG. 6A shows a front, perspective view of an exemplary wall mountbracket on the smart mirror.

FIG. 6B shows a front-side, flat representation of the wall mountbracket of FIG. 6A.

FIG. 6C shows a front-side view of the wall mount bracket of FIG. 6A.

FIG. 6D shows a side-side view of the wall mount bracket of FIG. 6A.

FIG. 7A shows a front, perspective view of an exemplary wall mountbracket on the wall side.

FIG. 7B shows a front-side, flat representation of the wall mountbracket of FIG. 7A.

FIG. 7C shows a front-side view of the wall mount bracket of FIG. 7A.

FIG. 7D shows a side-side view of the wall mount bracket of FIG. 7A.

FIG. 8A shows a front, perspective view of an exemplary of a safetybracket on the smart mirror side.

FIG. 8B shows a front-side, flat representation of the safety bracket ofFIG. 8A.

FIG. 8C shows a front-side view of the safety bracket of FIG. 8A.

FIG. 8D shows a side-side view of the safety bracket of FIG. 8A.

FIG. 9A shows a front, perspective view of an exemplary of a safetybracket on the wall side.

FIG. 9B shows a front-side, flat representation of the safety bracket ofFIG. 9A.

FIG. 9C shows a front-side view of the safety bracket of FIG. 9A.

FIG. 9D shows a side-side view of the safety bracket of FIG. 9A.

FIG. 10A shows a front-side view of an exemplary mirror glass.

FIG. 10B shows a front-side view of an exemplary safety film.

FIG. 10C shows a front-side view of the safety film of FIG. 10B alignedto the mirror glass of FIG. 10A.

FIG. 10D shows a front, perspective view of the safety film of FIG. 10B.

FIG. 11A shows a front, perspective view of an exemplary stand.

FIG. 11B shows another front, perspective view of the stand of FIG. 11A.

FIG. 11C shows a front-side view of the stand of FIG. 11A.

FIG. 11D shows a side-side view of the stand of FIG. 11A.

FIG. 11E shows a perspective view of the top bar of the stand of FIG.11A.

FIG. 11F shows a top-side view of the top bar of FIG. 11E.

FIG. 12A shows a front, perspective view of an exemplary upper displaypanel bracket.

FIG. 12B shows a front-side, flat representation of the upper displaypanel bracket of FIG. 12A.

FIG. 12C shows a front-side view of the upper display panel bracket ofFIG. 12A.

FIG. 12D shows a top-side view of the upper display panel bracket ofFIG. 12A.

FIG. 12E shows a side-side view of the upper display panel bracket ofFIG. 12A.

FIG. 12F shows a front-side, flat representation of the upper displaypanel bracket of FIG. 12A with various labeled holes for assembly.

FIG. 13A shows a front-side view of an exemplary antenna mountingbracket.

FIG. 13B shows a front, perspective view of the antenna mounting bracketof FIG. 13A.

FIG. 14A shows a perspective view of an exemplary camera mount.

FIG. 14B shows a front-side, flat representation of the camera mount ofFIG. 14A.

FIG. 14C shows a front-side view of the camera mount of FIG. 14A.

FIG. 14D shows a side-side view of the camera mount of FIG. 14A.

FIG. 14E shows a bottom-side view of the camera mount of FIG. 14A.

FIG. 15A shows a perspective view of an exemplary connector box with aL-bracket.

FIG. 15B shows a rear-side, flat representation of the connector box ofFIG. 15A.

FIG. 15C shows a rear-side view of the connector box with a L-bracket ofFIG. 15A.

FIG. 16A shows a perspective view of the connector box of FIG. 15A.

FIG. 16B shows a top-side view of the connector box of FIG. 16A.

FIG. 16C shows a bottom-side view of the connector box of FIG. 16A.

FIG. 16D shows a rear-side view of the connector box of FIG. 16A.

FIG. 16E shows a side-side view of the connector box of FIG. 16A.

FIG. 17A shows a perspective view of the L-bracket of FIG. 15A.

FIG. 17B shows a top-side view of the L-bracket of FIG. 17A.

FIG. 17C shows a rear-side, flat representation of the L-bracket of FIG.17A.

FIG. 17D shows a rear-side view of the L-bracket of FIG. 17A.

FIG. 17E shows a side-side view of the L-bracket of FIG. 17A.

FIG. 18 shows a wiring diagram for various components of an exemplarysmart mirror.

FIG. 19 shows an image of an exemplary biometric sensor worn on a user'swrist.

FIG. 20 shows an image of an exemplary biometric sensor worn around auser's ribcage.

FIG. 21 shows a flowchart of an exemplary method of setting up and asmart mirror in conjunction.

FIG. 22 shows a summary of the various wireless connections used tocommunicatively couple a smart mirror to a smart device.

FIG. 23A shows an exemplary icon displayed on the smart mirror toindicate the smart mirror is disconnected from the smart device.

FIG. 23B shows a pause notification displayed on the smart mirror whenthe application is closed, minimized, or the smart device enters sleepmode.

FIGS. 24A-1 through 24A-3 show a flowchart of an exemplary healingprocess when loading the application.

FIGS. 24B-1 through 24B-2 show a flowchart of an exemplary healingprocess when a connectivity break occurs during a workout.

FIGS. 24C-1 through 24C-3 show a flowchart of an exemplary healingprocess when a user accesses the application settings.

FIG. 25 shows a flowchart diagram describing how the smart mirror iscommunicatively coupled to another device via a Bluetooth Low Energyconnection.

FIG. 26A shows a flowchart describing an exemplary method of using asmart mirror with a HostAP mode.

FIG. 26B shows a flowchart for the ‘Attempt Connection’ process of FIG.26A.

FIG. 26C shows a flowchart describing an exemplary method of connectinga device with an iOS operating system to the smart mirror and/or settingup a network connection.

FIG. 27 shows an exemplary graphical user interface (GUI) on a smartphone.

FIG. 28A shows an exemplary GUI on the smart device to controlconnectivity of a smart device to a smart mirror and/or a networkconnection.

FIG. 28B show an exemplary GUI on the smart device showing notificationswhether the smart mirror is connected to the smart device.

FIG. 28C shows an exemplary GUI on the smart device to customize theuser interface of the smart mirror and/or the smart device.

FIG. 28D shows an exemplary GUI on the smart device to manage connectionof the smart device to various peripheral device such as an audio deviceor a biometric sensor.

FIG. 28E shows an exemplary GUI on the smart device to select a musicsource such as the local device or a third party service.

FIG. 28F shows an exemplary GUI on the smart device of music playlists.

FIG. 29A shows an exemplary GUI on the smart device to browse and selecta fitness class from a listing of fitness classes.

FIG. 29B shows an exemplary GUI on the smart device of filters used tonarrow down a listing of fitness classes.

FIG. 29C shows an exemplary GUI on the smart device of an exemplaryselection of a fitness class.

FIG. 30A shows an exemplary GUI on the smart device to control a fitnessclass played on the smart mirror.

FIG. 30B shows an exemplary GUI on the smart device of a workout log.

FIG. 30C shows an exemplary GUI on the smart device to provide userfeedback on an instructor and/or a fitness class.

FIG. 31A shows an exemplary GUI on the smart mirror of a fitness classoverview.

FIG. 31B shows an exemplary GUI on the smart mirror of an exemplary userinterface during a workout.

FIG. 31C shows an exemplary GUI on the smart mirror of a messagedisplayed to a user based on the user's biometric data.

FIG. 31D shows an exemplary GUI on the smart mirror of a messagedisplayed to a user showing adaptation of the workout based on the userpreferences.

FIG. 31E shows an exemplary GUI on the smart mirror of avatars of otheruser's in the same fitness class.

FIG. 31F shows an exemplary GUI on the smart mirror with the user'sheart rate displayed on a target heart rate zone.

FIG. 31G shows an exemplary GUI on the smart mirror with a messageindicating the user's heart rate meets a target heart rate zone.

FIG. 31H shows an exemplary GUI on the smart mirror with the user'sheart rate falls outside a target heart rate zone.

FIG. 31I shows an exemplary GUI on the smart mirror with the user'sscore and a target score.

FIG. 31J shows an exemplary GUI on the smart mirror with the user'sscore and a target score at a later period of time in the workoutrelative to FIG. 311.

FIG. 31K shows an exemplary GUI on the smart mirror with the user'sscore and a target score at a later period of time in the workoutrelative to FIG. 31J.

FIG. 31L shows an exemplary GUI on the smart mirror of a user's workoutlog.

FIG. 31M shows an exemplary GUI on the smart mirror of a user'sperformance after a particular workout.

FIG. 32A shows an exemplary GUI on the smart mirror notifying the usertake an image of themselves.

FIG. 32B shows an exemplary GUI on the smart mirror of the user's imageacquired by the camera of the smart mirror.

FIG. 32C shows an exemplary GUI on the smart mirror of multiple user'simages.

FIG. 33A shows an exemplary instructor user interface on a web browserwith a class schedule and an instructor dashboard of users attending theinstructor's class.

FIG. 33B shows an exemplary instructor user interface on a web browserof user information for a specific user in the class.

FIG. 33C shows an exemplary instructor user interface on a web browserof another class schedule.

DETAILED DESCRIPTION

The development of fitness-related technologies has been motivated, inpart, by a desire to provide a more convenient approach for receivingguided workouts that are not restricted by a predetermined, rigidworkout schedule and/or even do not involve going to a gym. For example,a pre-recorded, self-guided workout has been available for many years invarious formats (e.g., a video cassette tape, a digital versatile disc(DVD), a Blu-ray disc (BD), a video streamed using a streaming serviceand/or the Internet). The self-guided workout typically involves a videobeing played on a user's television and/or smart device (e.g., a smartphone, a tablet) where the user emulates the instructor's workout. Thisapproach provides greater convenience to the user (e.g., the user canplay a self-guided workout at home), but lacks the personalizationand/or individual guidance of a live instructor.

In another example, live workouts by an instructor may be streamed to auser using a streaming service connected to a user's smart device (e.g.,a smart phone, a tablet). However, live streaming a workout in thismanner typically provides a poor user experience and a mixed quality ofcontent due to: (1) the user sifting through a substantial amount ofpoor content, (2) the user being limited to either a location with atelevision screen to display the stream or to watching the stream on asmall display such as on a phone or a laptop, (3) the inability to trackthe user's progress in the workout, and (4) the lack of personalization.Some conventional exercise equipment may offer an integrated displaythat shows a live stream of an instructor. However, this equipment stillrestricts a user's ability to personalize a workout and/or to receiveindividual guidance from the instructor. Furthermore, the types ofexercise available to the user may be restricted to the exerciseequipment within which the display is integrated (e.g., an exercisebicycle may only provide workouts related to cycling).

Although boutique fitness studios may provide greater personalizedworkouts to the user, the boutique studio may still be difficult forusers to access for many of the same reasons as conventional gyms (e.g.,cost, schedule, location). Furthermore, many boutique fitness studiosprovide workouts primarily in a group setting, which may be undesirablefor many people due to personal preferences and/or physical limitations.

The present disclosure is thus directed to a reflective video displayapparatus (also referred to as a “smart mirror” and an “interactiveexercise system”) and methods for using the reflective video displayapparatus. The smart mirror includes an integrated display configured toshow a workout (a prerecorded video or a live stream) and an interfacethat enable a user to personalize a workout. Furthermore, the smartmirror may allow users and/or instructors to interact with each otherduring the workout (e.g., providing feedback to the instructor on thepace of the workout, correcting a user's form during a particularexercise routine) in a manner similar to a conventional workout at a gymor boutique fitness studio where the user and the instructor are in thesame room. j

The smart mirror may be coupled to a wired or wireless network that is,in turn, connected to a remote server, the Internet, and/or other smartmirrors to provide a user access to a selection of various workoutsavailable for download and/or streaming via an online or an app-basedcomponent. The user may select a workout based on several factorsincluding, but not limited to the user's exercise preferences, skilllevel, equipment availability, physical limitations such as injuries,desired movement pattern, and style. Unlike conventional exerciseequipment, the smart mirror may not include any additional equipmentand, hence, is not constrained to a particular type of exercise.However, the smart mirror may nonetheless be used in combination withother exercise equipment (e.g., a jump rope, exercise bike, treadmill,free weights, weight machines, exercise bars, and so on) depending onthe workout.

Once selected, the video exercise may then be shown on the display ofthe smart mirror. The smart mirror may be sufficiently reflective suchthat the user sees their own reflection in the smart mirror during theworkout, thus providing visual feedback to the user to evaluate theirmovement and form. Depending on the size and position of the smartmirror, the user's reflection may appear superimposed on an image of aperson in the video (e.g., a trainer). This allows the user to bettermatch his or her motion to the trainer's motion (e.g., for proper formor to learn a new exercise).

The video exercise may further be configured to show only the trainerand the workout equipment used during the workout. Additionally, thesmart mirror may show text or visual graphics of biometric data withoutany background shapes, images, or screens (e.g., a black background). Inthis manner, the smart mirror may better reflect the user and the user'senvironment unlike conventional devices, which typically suffer from anover-cluttered interface that obscures the user and the environment.Once again, the video exercise may be prerecorded or streamed by a livepersonal trainer in a private or group/class setting.

The smart mirror may include a camera to record a user during theworkout. The recorded video may be streamed to another person (e.g., aninstructor, another user) for live feedback or stored locally on thesmart mirror or another device (e.g., a server, a user's smart phone, auser's computer). In cases where video of the user is recorded during alive workout, the video captured by the smart mirror and any biometricfeedback data may be streamed to the instructor for real-timemonitoring. This may enable the instructor to provide personalizeddirection and adjustment to the user during the workout. For example, ina class-type setting, the trainer may monitor individual feeds of videoand biometric data from each user participating in the workout and mayselect a subset of the feeds to pay close attention to as the workoutprogresses. In cases where video of the user is stored, the recordedworkouts of the user may be replayed on the smart mirror or sharedonline with other users. The recorded video may thus be replayed as afuture workout for the user, compared to subsequent workouts by the useror other users, and/or evaluated to ascertain improvements to the user'sworkout performance.

The smart mirror may be coupled to a device that provides biometricfeedback of the user during the workout. The device may be a wearable orhandheld device (e.g., a heart rate monitor, a step monitor). Thebiometric feedback may be delivered to the smart mirror by a wired or awireless connection (e.g., Bluetooth low energy) directly to the deviceor to the device via the user's smart phone. The biometric data may bedisplayed on the smart mirror during the workout and may also be storedfor future comparison and/or analysis. The smart mirror may show thebiometric data in real-time in various formats including, but notlimited to the data by itself, the data with a target value set by theuser, another user, or the instructor, the data with historicalbiometric data acquired during previous workouts, or any combination ofthe foregoing. The smart mirror may include integrated storage (e.g., ahard disk drive, a solid state drive, random access memory) to store thebiometric data (and recorded video) permanently or temporarily. Thebiometric data (and recorded video) may also be uploaded for remotestorage (e.g., on a server, or a cloud-based system) via networkedconnection (e.g., the Internet).

Additionally, the biometric data recorded in previous workouts may beused to help select future workouts for the user. The recorded video ofthe user may also be processed as part of a biofeedback analysis todetect and analyze the movement of the user. This analysis may be usedto evaluate whether the user executed proper movement patterns duringeach exercise in the workout and to suggest areas for improvement infuture workouts with similar exercises.

The smart mirror may be shaped, dimensioned, and oriented to provide auser a reflection of their entire body for various movements and/orposes during the workout. The smart mirror may include a video displaypanel with a two-way mirror or two-way mirror film on top of the videodisplay panel. In this manner, the smart mirror is (fully) reflective inareas where the video display panel is not showing an image. The smartmirror may be mounted to a wall or configured to stand on a flatsurface. The smart mirror may further include communication componentsto facilitate connection to (1) biometric sensors used by the user, (2)internal communication to a user's smart device (e.g., a smart phone, atablet, or a computer), and/or (3) external communication to a remoteserver, a cloud, or the Internet.

The smart mirror may be used in a variety of settings including but notlimited to a home, hotel room, cruise ship, or other private or publicspaces. The smart mirror may also be used in conventional gymnasiumsand/or boutique fitness studios for individual or group exerciseprograms. Here, group exercise programs may include a group of users ina single studio or a group of users in one studio connected to otherindividual or groups of users in other studios/settings. The smartmirror may also be used in assisted living facilities, hospitals, orphysical therapy facilities to assist users with rehabilitation and/ormaintenance of their health.

While the smart mirror is described herein in the context of fitness andphysical therapy applications (e.g., online exercising, interactiveexercise, or interactive training), the smart mirror may be used moregenerally as a platform to provide users with interactive video content.Video content may include corresponding audio content as well. Forexample, the smart mirror may be used to provide other video contentincluding but not limited to cooking tutorials, lessons on arts andcrafts, home repair, car repair, and online educational courses. In somecases, the smart mirror may also be used as a television and/or smartdisplay for streaming content from a smart device. In this manner, thesmart mirror may replace conventional display devices and is designed tointegrate more seamlessly with the environment than a flat-screendisplay.

The concepts introduced above and discussed in greater detail below maybe implemented in numerous ways. Examples of specific implementationsand applications are provided primarily for illustrative purposes so asto enable those skilled in the art to practice the implementations andalternatives apparent to those skilled in the art.

The figures and example implementations described below are not meant tolimit the scope of the present implementations to a single embodiment.Other implementations are possible by way of interchange of some or allof the described or illustrated elements. Moreover, where certainelements of the disclosed example implementations may be partially orfully implemented using known components, in some instances only thoseportions of such known components that are useful for an understandingof the present implementations are described, and detailed descriptionsof other portions of such known components are omitted so as not toobscure the present implementations.

An Exemplary Smart Mirror

FIG. 1 shows an exemplary representation of a smart mirror 100. Thesmart mirror 100 may include a single board computer (SBC) 110 used tocontrol, in part, the operation of various subcomponents in the smartmirror 100 and to manage the flow of content to/from the smart mirror100 (e.g., video content, audio from the instructor or user, biometricfeedback analysis). The smart mirror 100 may include a display panel 120to show video content, a graphical user interface (GUI) from which theuser may interact and control the smart mirror 100, biometric feedbackdata, and/or other visual content. A camera 130 may be coupled to theSBC 110 to record a video and/or images of a user (e.g., while the useris exercising during a workout). An antenna 140 may be coupled to theSBC 110 to provide data transmission and/or reception between the smartmirror 100 and another device (e.g., a remote control device, abiometric sensor, a wireless router). The antenna 140 may comprisemultiple transmitters and receivers each tailored for a particularfrequency and/or wireless standard (e.g., Bluetooth, 802.11a, 802.11b,802.11g, 802.11n, 802.11ac, 2G, 3G, 4G, 4G LTE, 5G). An amplifier 150may be coupled to the SBC 110 to receive audio signals from the SBC 110for subsequent sound output through a left speaker 152 and/or a rightspeaker 154. A microphone array 160 may also be used to enable a user toinput voice commands and/or voice inputs to the smart mirror 100 (e.g.,to start/stop a workout, to talk to the instructor).The microphone array160 may also be coupled to the SBC 110 and include a digital signalprocessor (DSP).

A switched-mode power supply (SMPS) 170 may also be coupled to the SBC110 to supply and manage electrical power to the various components ofthe smart mirror 100 from an external electrical power supply system(e.g., a wall outlet). A switch 180 may be coupled to the SMPS 170and/or the microphone array 160 to switch the smart mirror 100 and themicrophone array 160 on and off. FIG. 18 shows electrical andlow-voltage differential signaling (LVDS) connections among thesecomponents and an LVDS power connection 1802 and signal connection 1804for the display panel 120.

The smart mirror 100 may also include additional components not shown inFIG. 1. For example, the smart mirror 100 may include onboard memory andstorage (nonvolatile and/or volatile memory) including, but not limitedto a hard disk drive (HDD), a solid state drive (SDD), flash memory,random access memory (RAM), and a secure digital (SD) card. This onboardmemory and/or storage may be used to store firmware and/or software forthe operation of the smart mirror 100. As described above, the onboardmemory and/or storage may also be used to store (temporarily and/orpermanently) other data including, but not limited to video content,audio, video of the user, biometric feedback data, and user settings. Inanother example, the smart mirror 100 may include a frame 200, describedin greater detail below with respect to FIG. 3B, to mount and supportthe various components of the smart mirror 100.

The smart mirror 100 may be deployed in an environment (e.g., a user'shome, a fitness studio) in several ways. For example, FIGS. 2A and 2Bshow the smart mirror 100 mounted to a stand 210 mounted to the bottomof the smart mirror 100. The smart mirror 100 reflects an image 229 ofthe user (here, taking a picture of the smart mirror 100 with a smartphone) and the surrounding environment. The smart mirror 100 also showsvideo content through a partially reflecting section 226, which blendsnearly seamlessly with a fully reflecting section 228 to reflect theuser's image 229 and the surrounding environment. The fully reflectingsection 228 has a dark background and the partially reflecting section226 is over a display panel 120 (FIG. 3B), which is dark when off toprovide a nearly seamless reflection under ambient lighting.

The stand 210 is used, in part, to position the smart mirror 100 at somedistance above the ground. The stand 210 may be used to support thesmart mirror 100 along a vertical orientation (e.g., a plane of thedisplay panel 120 is parallel to an adjoining wall). The stand 210 mayalso support the smart mirror 100 at a tilted orientation (defined by anangle relative to the wall) as shown in FIGS. 2A and 2B. The stand 210may include a high friction base (e.g., a rubber foot) to prevent thesmart mirror 100 from slipping along the floor when tilted. In somedesigns, the stand 210 may remain fixed relative to the frame 200 or mayallow for articulation of the frame 200 relative to the stand 210 aboutsome motion axis (e.g., a pivot axis). For instance, as the frame 200 isrotated, the stand 210 may remain unchanged in orientation and/orplacement with respect to the floor.

The smart mirror 100 may also be mounted to a wall directly, as depictedin FIG. 2C, or hung from a ceiling (not shown). Again, the smart mirror100 appears completely reflective when the display is off. When thedisplay is on, the display projects video imagery (e.g., of a trainer orexercise instructor) through the partially reflecting section 226 to theuser, who may see a reflected image 229 of herself superimposed on thevideo imagery in the partially reflecting section 226. A fullyreflecting section 228 bordering the partially reflecting section 226also reflect the user's image. And when the display is off, the smartmirror 100 simply appears to be a plain mirror.

The smart mirror 100 may also be supported by a free-standing standroughly vertically. Said in another way, the free-standing stand may siton the ground or another horizontal surface and hold the smart mirror100 so that it faces a user. The free-standing stand may be mounted tothe bottom, the side, and/or the rear of the smart mirror 100. Thefree-standing stand may include one, two, three, or more legs to providea stable platform for the smart mirror 100 (so the smart mirror 100 isunlikely to tip over). Each leg may have a high friction base (e.g., arubber foot) to prevent the stand from slipping. In some designs, atleast one leg may include a wheel to facilitate transport and/oradjustment of the smart mirror 100. Similar to the stand 210, thefree-standing stand may also allow the smart mirror 100 to be tiltedabout a pivot axis.

FIGS. 3A-3D show several views of an exemplary smart mirror 100 with thestand 210 described above. As shown, the smart mirror 100 may besubdivided into several assemblies corresponding to the componentsdescribed above. For instance, the smart mirror 100 may include a frame200 comprising an inner frame 202 and an outer shell 204. The innerframe 202 may be used as a chassis onto which the other componentsdescribed with reference to FIG. 1 are mounted to. The outer shell 204may be used, in part, as an exterior housing to protect the inner frame202 and the various components of the smart mirror 100 containedtherein. The smart mirror 100 may include a display panel 120 mountedinto the inner frame 202. The smart mirror 100 may include mirror glass220 disposed over the display panel 120 to provide reflections of theuser and the user's environment. The smart mirror 100 may also includevarious electronics separated into an upper electronics assembly 230disposed towards the top of the inner frame 202 and a lower electronicsassembly 240 disposed towards the bottom of the inner frame 202.

As shown in FIG. 3C, the upper electronics assembly 230 may include theantenna 140, the camera 130, the microphone 160, and the SBC 110. FIG.3D shows the lower electronics assembly 240 may include the SMPS 170,the switch 180, the amplifier 150, and the speakers 152 and 154.Additionally, the smart mirror 100 may include the stand 210 disposed onthe bottom of the inner frame 202.

The smart mirror 100 in FIGS. 3A-3D represents one exemplary size andaspect ratio. The smart mirror 100, however, may generally be larger orsmaller in size and/or have various aspect ratios. For example, a largersmart mirror 100 may be used to accommodate a taller user and/ormultiple users. A smaller smart mirror 100 may be used to accommodateshorter users and/or to increase portability. Generally, the smartmirror 100 may have a height from about 24 inches to about 96 inches anda width from about 9 inches to about 120 inches. The aspect ratio of thesmart mirror 100 may thus vary according to the respective ranges of theheight and the width disclosed. FIGS. 4A-4E show several views of anexemplary inner frame 202. The inner frame 202 may be dimensioned andshaped to have an interior cavity within which contains the variouscomponents of the smart mirror 100 such as the SBC 110, the displaypanel 120, the camera 130, the antenna 140, the amplifier 150, thespeakers 152 and 154, the microphone array 160, the SMPS 170, and theswitch 180. The inner frame 202 may also include several mounting points(as indicated in FIG. 4E as holes A through H) to mount theaforementioned components to the inner frame 202 using various couplingmembers including, but not limited to screw fasteners, bolt fasteners,snap fit connectors, and adhesive. The inner frame 202 may also includeapertures through which the camera 130 and the microphone 160 may recordvideo and receive sound, respectively, from the user in the environment.

FIGS. 5A-5F show several views of an exemplary outer shell 204. Theouter shell 204 may surround, at least in part, the inner frame 202. Forexample, FIG. 3B shows the outer shell 204 has an interior cavity thatmay contain therein the inner frame 202. The outer shell 204 may be usedprimarily to protect the inner frame 202 and the components containedtherein. The outer shell 204 may include a plurality of ventilationholes or perforations to facilitate cooling of the various electroniccomponents in the smart mirror 100. The outer shell 204 may also includea plurality of openings to transmit sound from the speakers 152 and 154to the user. The outer shell 204 may also include an opening throughwhich a port on a connector box is used to receive electrical power.

As shown in FIG. 2C, the smart mirror 100 may also be directly mountedto a wall for deployment. Various wall mounting mechanisms may be usedincluding, but not limited to corresponding hooks on the wall and thesmart mirror 100, a mounting bracket fastened to the wall and the smartmirror 100 via screw or bolt fasteners, and adhesive tape.

FIG. 3C shows an exemplary hook mechanism using a mounting bracket 302 aon the smart mirror 100 and a corresponding mounting bracket 302 b to beattached to the wall. FIGS. 7A-7D show additional views of the mountingbracket 302 b. As shown, the mounting brackets 302 a and 302 b may havea width substantially similar to the width of the outer shell 204 toprovide greater stability when hanging the smart mirror 100 from themounting bracket 302 b. As shown, the mounting bracket 302 a may becoupled to the outer shell 204 using the same mounting points to mountan upper display panel bracket 304 a disposed within the inner frame202. The mounting bracket 302 b may include multiple holes and/or slotsto facilitate attachment to the wall. FIGS. 6A-6D show additional viewsof the mounting bracket 302 a.

FIG. 3C shows a mirror-side safety bracket 306 a and a wall-side safetybracket 306 b. FIGS. 8A-8D show more views of the mirror-side safetybracket 306 a. FIGS. 9A-9D show more views of the wall-side safetybracket 306 b. These safety hooks 306 a and 306 b prevent the smartmirror 100 from tipping over when the smart mirror 100 is mounted to thestand 210. Similar to the wall mounting brackets 302 a and 302 bdescribed above, the safety hook may also comprise a safety bracket 306a mounted to the outer shell 204 of the smart mirror 100 and acorresponding safety bracket 30 b mounted to the wall. As shown in FIG.3C, the safety brackets 306 a and 306 b may be positioned near a centerline (e.g., a vertical axis) of the smart mirror 100 to increasestability.

FIG. 3C shows both the safety brackets 306 a and 306 b and the wallmounting brackets 302 a and 302 b. This is for showing where theserespective components are placed with respect to the other components ofthe smart mirror 100. In practice, the smart mirror 100 may use just thesafety brackets 306 a and 306 b or the wall mounting brackets 302 a and302 b, but not both together.

The inner frame 202, the outer shell 204, the mounting brackets 302 aand 302 b, and the safety brackets 306 a and 306 b may be formed ofvarious materials including, but not limited to steel, aluminum,fiberglass, carbon fiber, polyethylene terephthalate glycol (PETG), andplastic. For example, the inner frame 202, outer shell 204, mountingbrackets 302 a and 302 b, and safety brackets 306 a and 306 b may beformed by patterning a flat sheet of metal, bending the sheet into thedesired three-dimensional shape, and welding adjoining edges to form thefinished component. Additional coatings (e.g., powder coatings, paint)may be applied to the inner frame 202, outer shell 204, mountingbrackets 302 a and 302 b, and safety brackets 306 a and 306 b to reduceenvironmental contamination and/or for aesthetics.

The mirror glass 220 may be a two-way mirror or a two-way mirror filmdisposed on or in front of the display panel 120. The mirror glass 220may thus be semi-reflective and semi-transparent to visible light. Themirror glass 220 may be substantially reflective when the display panel120 is not active or in regions of the display panel 120 that showdarker colors. The mirror glass 220 may be substantially transparent inregions of the display panel 120 that show brighter colors. Said inanother way, the mirror glass 220 may appear reflective to the user whenthe intensity of light reflected by the mirror glass 220 (e.g.,environmental light, natural light, light reflected off objects or theuser in the environment) is greater than the intensity of lighttransmitted through the mirror glass 220 (e.g., light emitted by thedisplay panel 120). Conversely, the mirror glass 220 may appeartransparent to the user when the intensity of light reflected by themirror glass 220 is less than the intensity of light transmitted throughthe mirror glass 220.

The mirror glass 220 may be coupled to the inner frame 202 using variouscoupling mechanisms including, but not limited to a tape, an adhesive, aclamp, a snap fit connector bonded to the mirror glass 220, and a screwfastener or a bolt fastener via a tab or pin bonded to the mirror glass220. A safety film 222 may be attached directly to the mirror glass 220to prevent the mirror glass 220 from shattering: if the mirror glass 220breaks, the pieces of broken glass would remain affixed to the safetyfilm 222. The safety film 222 may be transparent and can be patterned orprinted with opaque (black) regions. For instance, the safety film 222may be transparent over the partially reflecting section 226 of thesmart mirror 100 and opaque over the fully reflecting section 228 of thesmart mirror 100. Additionally, the safety film 222 may not fully coverthe surface of the mirror glass 220. The patterning of the safety film222 may be tailored to create a seamless appearance between the displaypanel 120 and the remaining portion of the mirror glass 220 when thesmart mirror 100 is viewed from the front.

Double-sided adhesive tape 224 may be used to attach the mirror glass220 to the inner frame 202 within the frame 200, as shown in FIG. 3B. Onone side, the adhesive tape 224 is attached directly to the surface ofthe inner frame 202. On the other side, the adhesive tape 224 isattached to the safety film 222 or to the exposed mirror glass 220.

The mirror glass 220 may also be removable from the frame 200 afterinstallation to allow replacement of the mirror glass 220 (as opposed tothe entire smart mirror 100) in the event the mirror glass 220 isdamaged. This may be accomplished by bonding the mirror glass 220 andsafety film 222 to a set of pins or tabs that fit into a correspondingset of holes or slots in the frame 200. The number and distribution ofpins and/or tabs may be tailored to reduce stress concentrations on themirror glass 220 when assembled. The pins or tabs may be coupled to theframe 200 using a coupling member including, but not limited to a screwfastener, a bolt fastener, and a snap fit connector. The coupling memberis configured to securely mount the mirror glass 220 to the frame 200but may also allow a user to subsequently disassemble the smart mirror100 to remove/replace the mirror glass 220.

The mirror glass 220 may be formed from various materials including, butnot limited to glass, acrylic, mylar, plexiglass, a thermoplastic,polymethyl methacrylate (PMMA), or any other materials transparent tovisible light. The reflective properties of the mirror glass 220 may bemodified by a coating disposed by a partially reflective coating formedof various materials including but not limited to aluminum, silver, anddielectric coatings (e.g., a Bragg mirror). The safety film 222 may beformed from a flexible thin film polymeric material. The double-sidedadhesive tape 224 may be various types of adhesive tapes including, butnot limited to a very high bonding (VHB) tape, an ultra-high bonding(UHB), and an acrylic foam tape (AFT).

The smart mirror 100 may also be stylistically reconfigurable. Forexample, the smart mirror 100 may appear float when mounted to a wall.In this configuration, the edges of the mirror glass 220 may be fullyexposed. The mirror glass's lateral dimensions may be equal to or largerthan the lateral dimensions of the frame 210 located behind the mirrorglass 220 as shown in FIGS. 10A-10D. As described above, the frame 200may contain therein the various components of the smart mirror 100(e.g., the SBC 110, the display panel 120, the camera 130, the antenna140, the amplifier 150, the speakers 152 and 154, the microphone array160, the SMPS 170, and the switch 180). Thus, a user directly facing thefront of the smart mirror 100 may be unable to observe the frame 200located behind the mirror glass 220, giving the impression that thesmart mirror 100 is floating in space parallel to the wall.

Conventional floating mirror displays are typically two-piece assemblieswhere the mirror glass is positioned in front of the display. The mirrorglass is typically hung from an elevated position, such as a ceiling orwall, and positioned in front of the display, which may also be hungfrom the ceiling or wall. This two-piece assembly increases installationcomplexity and limits the conventional mirror displays to environmentswhere such mounting points are available. Other types of conventionalmirror displays may be assembled in a framed configuration where a frontside frame and a back side frame are joined together to hold the mirrorglass in place. For this configuration, the edges of the mirror glassmay be obscured by the front side frame and the front side frame may beobserved by a user, thus affecting the aesthetic quality of the floatingmirror configuration.

In contrast, the smart mirror 100 described herein may be constructedsuch that the mirror glass 220 is attached to a frame 200 thus forming aone-piece assembly. In the exemplary smart mirror 100 shown in FIG. 3A,the mirror glass 220 is bonded to a safety film 222. The safety film222, in turn, is bonded to the inner frame 202 with double-sidedadhesive tape 224. In this manner, the mirror glass 220 may be attachedto the frame 200 such that the smart mirror 100 appears to float inspace when mounted directly to a wall.

The smart mirror 100 may also allow for a decorative frame to be mountedon the front and/or side of the smart mirror 100. The decorative framemay be coupled to the outer shell 204 of the frame 200 located behindthe mirror glass 220. The decorative frame may be coupled to the outershell 204 using one or more coupling members including, but not limitedto a screw fastener, a bolt fastener, and a snap fit connector. Thedecorative frame may also be coupled to the outer shell 204 using one ormore magnets, thus increasing the ease of installation and reducing theassembly time. In some designs, the smart mirror 100 may include adecorative frame mounted on the edges of the mirror glass 220. If a userwishes to replace the decorative frame, the user may disassemble thesmart mirror 100 using the pins or tabs described above to replace themirror glass 220.

FIGS. 11A-11F show additional views of the stand 210 may be used tosupport the smart mirror 100 in a substantially vertical orientation(with or without tilt). The stand 210 is comprised of a U-shaped bracket214 disposed beneath the inner frame 202 and the outer shell 204. TheU-shaped bracket 214 is joined to a top bar 216. As shown, the U-shapedbracket 214 and the top bar 216 may be shaped and dimensioned to conformto the outer shell 204, thus providing a continuous surface around thesides of the smart mirror 100. The top bar 216 may include openings forthe speakers 152 and 154. The top bar 216 may also include an openingthrough which the switch 180 may be accessed by the user. The stand 210may also include a high friction base 212 (e.g., rubber feet) disposedon the bottom of the U-shaped bracket 214 as shown in FIG. 3B. The highfriction base 212 may be used to prevent the smart mirror 100 fromslipping along the floor, especially when the smart mirror 100 ispartially tilted. The U-shaped bracket 214 and the top bar 216 may beformed of various materials including, but not limited to steel,aluminum, fiberglass, carbon fiber, polyethylene terephthalate glycol(PETG), and plastic.

The height of the mirror glass 220 and/or the display panel 120 of thesmart mirror 100 may also be adjustable to accommodate different userswith different heights. The smart mirror 100 may be designed to have aparticular height range to accommodate a majority of users. If a userfalls outside of this height range, the smart mirror 100 may be adjustedaccordingly. Height adjustment may be accomplished in several ways. Inone example, a slot-rail mechanism may be integrated into the smartmirror 100 using the frame 200 and the stand 210. For instance, theframe 200 may incorporate at least one slot between the outer shell 204and the inner frame 202 to accommodate a rail on the stand 210. The railon the stand 210 may thus be slidably adjustable along the slot in theframe 200. A locking mechanism may be included to secure the rail to theslot at a desired position. The locking mechanism may come in variousforms including, but not limited to a ratcheting mechanism that allowsmotion along one direction (e.g., extension of the rail relative to theslot) and a release mechanism to allow motion in the opposite direction,a plurality of holes in the slot (or rail) with a spring-mounted pin inthe rail (or slot) for securement, a clamping mechanism (e.g., a lockingtab) to hold the rail against the slot via a frictional force.

The electronic components of the smart mirror 100 may be disposed invarious locations on the inner frame 202 and the outer shell 204. In theexemplary smart mirror 100 shown in FIGS. 3A-3D, the electroniccomponents are disposed primarily in the upper electronics assembly 230and the lower electronics assembly 240 to simplify assembly and placerespective components in preferred locations on the smart mirror 100with respect to where a user may be located. Any wiring to electricallycouple the electronic components may be routed along the inner frame 202and/or the interior cavity of the outer shell 204.

The display panel 120 is primarily used to show video content to theuser. The display panel 120 may be various types of displays including,but not limited to a liquid crystal display (LCD), a light emittingdiode (LED) display, an organic light emitting diode (OLED) display. Thedisplay panel 120 may be configured to emit a lower light intensity whendisplaying darker colors (or may even turn off the respective pixels) inorder to enhance the performance of the two-way mirror glass 220. Insome instances, the display panel 120 may also be touch sensitive toprovide additional interactive control of the smart mirror 100 to theuser. The touch sensitivity may be calibrated to account for the mirrorglass 220 and any other intermediate components (e.g., the safety film222) disposed between the display panel 120 and the environment.

As shown in FIGS. 3C and 3D, the display panel 120 may be mounted to theinner frame 202 using an upper display panel bracket 304 a and a lowerdisplay panel bracket 304 b. FIGS. 12A-12F show additional views of theupper display panel bracket 304 a. The upper display panel bracket 304 aand the lower display panel bracket 304 b may be coupled to the innerframe 202 and the display panel 120 respectively using various couplingmechanisms including, but not limited to screw fasteners, boltfasteners, snap fit connectors, or adhesive. The upper display panelbracket 304 a and the lower display panel bracket 304 b may be formed ofvarious materials including, but not limited to steel, aluminum,fiberglass, carbon fiber, polyethylene terephthalate glycol (PETG), andplastic.

The antenna 140 may comprise multiple antennas that each function as areceiver and/or a transmitter to communicate with various externaldevices, such as a user's smart device (e.g., a computer, a smart phone,a tablet), a biometric sensor (e.g., a heart rate monitor), and/or aremote server or cloud server to stream or play video content. Onceagain, the antenna 140 may conform to various wireless standardsincluding, but not limited to Bluetooth, 802.11a, 802.11b, 802.11g,802.11n, 802.11ac, 2G, 3G, 4G, 4G LTE, and 5G standards.

FIGS. 13A and 13B show an antenna mounting bracket 1300 that can be usedto hold the antenna 140 in place. The antenna mounting bracket 1300 maybe formed of various materials including, but not limited to steel,aluminum, fiberglass, carbon fiber, polyethylene terephthalate glycol(PETG), and plastic.

The microphone 160 may be used to record a user's voice and/or otherambient sounds. The microphone 160 may include a muffler to reduceunwanted ambient noise (e.g., a fan or street noise) from being acquiredby the smart mirror 100. Similar to the camera 130, audio recorded bythe microphone 160 may be shared with another person (e.g., aninstructor or another user) in real-time or recorded for later playback.In one aspect, the audio may be acquired with the video of the userusing timestamps that ensure the audio is synced to the video. Themicrophone 160 may also be coupled to the amplifier 150 to control thesound output from the speakers 152 and 154. For example, when a userspeaks, the microphone 160 may send a signal to the amplifier 150 toreduce the sound output from the speakers 152 and 154 to avoid unwantedaudio feedback. The microphone 160 may be used, in part, to enable voicecontrol of the smart mirror 100. For example, a user mayactivate/deactivate the smart mirror 100 and navigate, start, and stopworkouts with their voice.

The left and right speakers 152 and 154 may be used to output sound tothe user (e.g., instructions from the instructor, music, sound effects).The speakers 152 and 154 may be low profile and configured to emit soundin one or more desired frequency bands. In some designs, the speakers152 and 154 may be oriented to emit sound in a direction transverse tothe front of the mirror glass 220 (e.g., towards the floor or theceiling) to reduce the thickness of the smart mirror 100 as shown inFIG. 3B. In some designs, the speakers 152 and 154 may be oriented toemit sound in a direction towards a user located in front of the mirrorglass 220. In this configuration, sound may be emitted through openingson the front of the smart mirror 100 (e.g., openings in the mirror glass220). The mirror glass 220 may also vibrate with the speakers 152 and154 to produce sound emitted towards the user. The mirror glass 220 maythus be tuned to emit sounds at frequencies that may be otherwiseunavailable in a front-facing, low-profile speaker. The sound output maybe controlled, in part, by the amplifier 150.

As described above, the camera 130 in the smart mirror 100 may be usedto acquire video and/or still images of the user while the user performsan activity (e.g., a workout). The video of the user may then be sharedwith an instructor to allow the instructor to observe and provideguidance to the user during a workout. The video may also be shared withother users of other smart mirrors for comparisons or competition. Thevideo of the user may also be shown on the display panel 120 inreal-time or stored for subsequent playback. For example, the video ofthe user may be used for self-evaluation during or after a workout byproviding a visual comparison of the user to the instructor. Storedvideo may also allow user to evaluate their progress or improvement whenperforming similar exercises over time.

The video may also be processed, in real-time during a workout or aftera workout is finished, to derive biometric data of the user based on themovement and motion of the user. For example, image analysis techniquesmay be used to determine various aspects of a user's workout including,but not limited to a user's breathing rate as a function of time, auser's performance in reproducing a proper form or motion of aparticular exercise, the number of repetitions performed by the userduring a workout, stresses on a user's limbs or joints that may lead toinjury, and a user's stamina based on deviations of a particularexercise over time. This biometric data derived from the video may beused in combination with biometric data acquired by a biometric sensorworn by the user to provide a user additional analysis on their workout.

The camera 130 may be one of several cameras mounted in or on the smartmirror 100, where each camera may be configured to image differentaspects of a user. The camera 130 may include a standard web camera. Inanother example, the camera 130 may include a spatial motion sensingcamera (e.g., a Microsoft Kinect) capable of tracking a user's motionwithin a three-dimensional (3D) space. The motion sensing camera mayhave sufficient spatial resolution to track individual extremities(e.g., arms, legs, hands, feet, fingers, toes). This data may be used toreconstruct a high fidelity 3D model of the user that is animated inaccordance to the user's movement as a function of time. The 3D model ofthe user may thus provide additional information to the user and/or theinstructor to assess a user's performance in executing a workout and toguide the user on proper form and technique. For example, the 3D modelmay be displayed to the user with a comparison to a second 3D model ofanother person (e.g., the instructor) performing an exercise with acorrect form and technique. The motion sensing camera may also be usedto identify and track the motion of multiple users.

In yet another example, the camera 130 may include a thermal camera(e.g., a forward-looking infrared (FLIR) camera) to generate temperaturemaps of the user's skin. These temperature maps may be used to trackspatial and temporal changes to a user's skin temperature during aworkout (e.g., resting, exercising, recovering after a workout), whichmay provide additional biometric data such as a user's hydration levels.

FIGS. 14A-14E show several views of an exemplary camera mount 1400 thatcan be used to mount the camera 130 to the inner frame 202. The cameramount 1400 may be a mechanical component designed to orient the camera130 such that the field of view of the camera 130 captures the userunder most use cases. For example, FIG. 3C shows the camera mount 1400has bent shape to tilt the camera 130 downwards. The camera mount 1400may be formed of various materials including, but not limited to steel,aluminum, fiberglass, carbon fiber, polyethylene terephthalate glycol(PETG), and plastic.

The smart mirror 100 may be configured to receive electrical inputs froman alternating current (AC) source or a direct current (DC) source. TheSMPS 170 may be compatible with both AC and DC input sources. The SMPS170 may be used, in part, to convert the electrical input into a desiredform for subsequent dissemination to other components of the smartmirror 100. For example, the SMPS 170 may be used to convert AC to DC orDC to AC. The SMPS 170 may also be used to adjust the voltage and/orcurrent of the input to desired values (e.g., increase the voltage from120 V to 240 V, decrease the voltage from 240 V to 120 V).

The SMPS 170 may be configured to receive electrical inputs via a powercord coupled to the smart mirror 100 or a wireless power transfer system(e.g., the smart mirror 100 has a receiver that receives wireless powerfrom a transmitter mounted to a wall via an inductive or capacitivecoupling mechanism). In one example, a standard InternationalElectrotechnical Commission (IEC) cable may be used to connect the SMPS170 directly to a standard wall outlet (e.g., a 120-240V/60 Hz outlet).In some designs, the SMPS 170 may be partially or wholly disposedoutside the frame 200 of the smart mirror 100 (e.g., an AC adapter for alaptop) to reduce the overall size of the smart mirror 100.

The smart mirror 100 may also include a battery (not shown) to providegreater portability. Thus, the deployment of the smart mirror 100 may beless constrained by the location of a power source (e.g., a wall outlet)within a particular room. The battery may be various types ofrechargeable or disposable batteries including, but not limited to alithium-ion battery, a nickel cadmium battery, and a nickel metalhydride battery. A rechargeable battery may be charged by connecting thesmart mirror 100 to a power source (e.g., connecting an IEC cable to awall outlet). The smart mirror 100 may also be configured to operatewhile charging.

The smart mirror 100 may also be turned on or off using a switch 180disposed on the smart mirror 100. The smart mirror 100 may also beactivated or deactivated remotely using another remote control device,such as a computer, a smart phone, or a tablet connected to the smartmirror 100.

FIGS. 15A-17E show several views of an exemplary connector box that mayalso be incorporated into the smart mirror 100. The connector box may beused to house wiring and wiring ports to connect the smart mirror 100 toan external power source (e.g., via an IEC cable). The connector box mayalso house wiring between the SMPS 170 and other electronic componentsin the smart mirror 100. The connector box may be formed of variousmaterials including, but not limited to steel, aluminum, fiberglass,carbon fiber, polyethylene terephthalate glycol (PETG), and plastic.

The smart mirror 100 may also include additional connectors to connectthe smart mirror 100 to other devices, such as a smart phone or atablet. For example, FIG. 3C shows the smart mirror 100 may include aUSB connector 310 disposed towards the top of the smart mirror 100. Insome applications, this connector may be used to provide a wiredconnection for software updates, firmware updates, and diagnostic testsof the smart mirror 100. The connector may also be used to transferpower from the smart mirror 100 to another device (e.g., charging asmart phone).

Biometric Devices and the Smart Mirror

A biometric sensor worn by the user may also be communicatively coupledto the smart mirror 100 to provide biometric data of the user during aworkout. As described below, the smart mirror 100 may display raw and/orprocessed biometric data to the user via the display panel 120. Thisbiometric data may be used for subsequent analysis to evaluate theuser's overall health and for recommending subsequent workouts to theuser. The biometric data may also be used to compare a user's activitylevel or performance to that of other users. Various types of biometricdata may be measured by one or more biometric sensors including, but notlimited to a user's heart rate, a user's step count, the motion of theuser's various extremities, the user's skin temperature, and the user'sperspiration rate. (A user may operate the smart mirror 100 withoutwearing any biometric sensor, in which case biometric data normallyacquired and displayed to the user may be replaced by a blank or adashed mark on the display panel 120 indicating that no biometric datais being acquired.)

The biometric sensor may be worn by the user in various ways. Forexample, FIG. 19 shows the user wearing a biometric sensor 1900 on herwrist. FIG. 20 shows another example of the user wearing a biometricsensor 2000 around her waist. A user may wear multiple biometricsensors, which, in some instances, may be tailored to measure certainbiometric data at certain locations on the user's body. Either biometricsensor may be coupled to the smart mirror 100 wirelessly using variouscommunication protocols including, but not limited to Bluetooth,802.11a, 802.11b, 802.11g, 802.11n, and 802.11ac, either directly or viaa smart phone or wireless router.

Using a Smart Mirror

The smart mirror 100 may be coupled to various devices and controlled,in part, using these devices. For example, the smart mirror 100 may beconnected to a smartphone, a smartwatch, a tablet, a dedicated remotefor the smart mirror 100, a smart exercise equipment (e.g., a treadmill,an exercise bike, a smart dumbbell), or a personal computer. Thesedevices may be networked and/or a web-enabled and thus used to accessvarious fitness-based features in a software application configured towork with the smart mirror 100 (e.g., an app for a Google Android, anApple iOS, or Microsoft Windows device).

The smart mirror 100 may also be used without connection to any device.For example, a user may control the smart mirror 100 using voice controlvia the microphone 160. The smart mirror 100 may also be controlledusing gesture commands in cases where the camera 130 includes a motionsensing camera or by applying image analysis techniques to video of theuser acquired by the camera 130. The smart mirror 100 may also becontrolled using touch commands in cases where the display panel 120 istouch sensitive.

FIG. 21 shows an exemplary method 2100 of using the smart mirror 100generally comprising a setup process 2101 followed by a use process2102. The setup process 2101 may be comprised of the following steps:(2104) turning on the mirror, (2108) downloading the app on a user'ssmart device, (2112) syncing the smart device to the smart mirror 100via an access point, (2116) syncing the smart mirror 100 to a network,and (2120) creating an account and filling out an onboarding survey. Theuse process 2102 may be comprised of the following steps: (2124)selecting and entering a workout, (2128) displaying the workout on thesmart mirror 100, (2132) collecting the workout progress and biometricdata from a biometric sensor, (2136) displaying post workout statisticsafter the workout ends, (2140) posting a workout survey, and (2144)adding the workout data to a user's fitness progress. Following the useprocess 2102, the smart mirror 100 may include the step of (2148)querying the user to select and enter another workout, which ifselected, leads to a repeat of steps (2128) through (2148). Thefollowing description elaborates upon the various steps shown in FIG.21.

Connectivity Between the Smart Mirror and Other Devices

As described above, the smart mirror 100 may be connected to variousdevices during operation. To maintain operation, the connection betweenany pair of devices (including the smart mirror 100) should bemonitored. The connection between a specific pair of devices may berepresented as a “connection state.” Thus, multiple connection statesmay be monitored during use of the smart mirror 100.

FIG. 22 shows a partial summary of connectivity options and statesbetween a smart mirror and one or more other devices. These connectionsinclude but are not limited to between: (1) a user's smart device (e.g.,the client) and the smart mirror 100 (e.g., the server), which may bemonitored using a WebSocket protocol and may have values of ‘connected’and ‘closed’, (2) a user's smart device and a Bluetooth low energy(BTLE) device, which may have values of ‘connected’ and ‘disconnected’;(3) a Bluetooth audio device (e.g., the client) and the smart mirror 100(e.g., the server), which may have values of ‘paired-connected’,‘paired-disconnected’, and ‘unpaired’; (4) a user's smart device and anapplication programming interface (API) server, which may be incommunication via a hypertext transfer protocol (HTTP) where theconnection is configured to be on demand with a request/responsestructure such that data is sent as a request and the server respondswith data; (5) the smart mirror 100 and an API server, which may be incommunication via HTTP where the connection is configured to be ondemand with a request/response structure such that data is sent as arequest and the server responds with data; (6) the smart mirror 100 anda streaming service, which may be in communication via a HTTP livestreaming (HLS) protocol and may have values of ‘connected/streaming’,‘buffering’, and ‘disconnected’, and (7) the smart mirror 100 and apublish/subscribe service, which may be in communication via Websocketand may have values of ‘connected’ and ‘disconnected.’

In the event that one device (e.g., the smart mirror 100, a smartdevice, a biometric sensor, a server, a network router) is disconnectedfrom another device, a healing process is used to re-establish andmaintain connection between the devices (also referred to herein as“device healing”). The healing process should preferably be transparentto the user such that when a connectivity problem arises, the workout isnot interrupted.

Generally, a user may connect a smart device (e.g., a smart phone ortablet) to the smart mirror 100 and a biometric sensor (e.g., a heartrate monitor (HRM) via Bluetooth) when installing and/or configuring thesmart mirror 100. The user may also connect other devices, such as aBluetooth audio device (e.g., a speaker or a microphone). The smartmirror 100, biometric sensor, and other accessory devices may be treatedas three distinct categories of devices from the perspective of theuser's smart device. The smart mirror 100 may automatically determinethe points in time and the appropriate length of time for the user'ssmart device to attempt to connect with the previously paired smartmirror 100 and the Bluetooth devices. This process should preferably beperformed without using excessive amounts of the smart device's battery.

Before a healing process is attempted, the user should pair their smartdevice to at least one device in a particular category via a “settings”interface on the application installed on the user's smart device. Thus,a healing process should only be attempted when (1) no devices in aparticular category are connected and (2) at least one device waspreviously paired device is present, but disconnected. The display panel120 of the smart mirror 100 may show an icon when the smart device isdisconnected from the smart mirror 100. An exemplary icon is shown inFIG. 23A, which may be displayed at the top right of the smart mirror100. When the app is closed, minimized, or the smart device enters asleep mode, the workout may be paused as shown in FIG. 23B.

FIGS. 24A-1 through 24A-3, 24B-1 through 24B-2, and 24C-1 through 24C-3show exemplary healing processes for situations where the user isloading the application on their smart device, a connectivity breakoccurs during a workout, and a user accesses the application settings,respectively. FIGS. 24A-1 through 24A-3, the healing process 2400 a maybe configured such that the user's smart device attempts to connect tothe smart mirror 100 and any previously paired Bluetooth devices (e.g.,a biometric sensor or an audio device). This healing process 2400 a maybe aborted after a scan time of about 30 seconds followed by a messageto the user indicating failure to connect to a device. If the smartmirror 100 and/or other devices are discovered, the user's smart deviceshould automatically connect to these devices. In the event more thanone Bluetooth device is found, the healing process 2400 a may connect tothe most recently found device and/or may allow the user to selectbetween multiple connected devices. The healing process 2400 a may berepeated under other conditions, such as when the user is opening apreview of a workout (e.g., Workout Preview) in the application on theirsmart device. Again, the healing process 2400 a may run for about 30seconds before the connection attempt is aborted.

FIGS. 24B-1 through 24B-2 show a healing process 2400 b for situationswhere any one of the devices described above is disconnected from theuser's smart device during a workout. For instance, when the user opens“Workout Options,” which include several settings for configuring theworkout, the healing process 2400 b may be performed if a previouslypaired device other than the smart mirror 100 is no longer connected tothe user's smart device. Again, this process may run for about 30seconds before the connection attempt is aborted. If a user's smartdevice disconnects from the smart mirror 100 and/or the biometric sensorduring a workout, the workout should preferably continue on the smartmirror 100 unless the user is requested to pause the workout. Once theconnection state between the user's smart device and the smart mirror100 and/or the biometric sensor is healed, the timers and workouts onthe user's smart device should sync to the smart mirror 100. In thismanner, the smart mirror 100 should dictate where the workout userinterface and/or the user's smart device should jump to once healingoccurs. After the workout is finished, the user's profile should beupdated with any relevant workout data regardless of whether aconnectivity issue occurred. If a user's smart device disconnects fromanother device, such as a Bluetooth audio device, the smart mirror 100should output audio in the interim until the audio device is healed.Again, this process may run for about 30 seconds before the connectionattempt is aborted.

FIGS. 24C-1 through 24C-3 show a healing process 2400 c configured forsituations where the user opens the “Settings” interface on the smartdevice. The healing process 2400 c may occur when a previously connectedBluetooth device or the smart mirror 100 is no longer connected. Again,this process may run for about 30 seconds before the connection attemptis aborted.

Data Transfer using Bluetooth

The smart mirror 100, the user's smart device, and/or other Bluetoothconnected devices may transfer data in a serial manner (e.g., from aclient to a server, from a server to a client) using various wirelesstechnologies, such as Bluetooth Low Energy. Depending on the wirelesstechnology used, a scheme may be devised for various aspects of datatransfer including, but not limited to an initial connection setup,chunking of messages for transfer, message reassembly, and connectionteardown. In the case where Bluetooth Low Energy is used, the scheme maybe used with or without Bluetooth Low Energy security features.

FIG. 25 shows an exemplary process 2500 to connect and transfer dataover Bluetooth Low Energy. Bluetooth Low Energy was included to enablefor low energy transfer of finite amounts of data. Many data profilesare ubiquitously supported by major implementations, but these dataprofiles typically do not support the transfer of stream data.Generally, stream data is sent over a SPP (Serial Port Profile)connection, but in some situations these connections may be unsupportedby the device manufacturer and/or may only be allowed by the devicemanufacturer under specific use cases due to excess consumption ofpower, which limits the use of such connections for practical use.Instead, the smart mirror 100 may use Bluetooth Low Energy using GenericAttributes (GATT) characteristics as the client-server communicationprotocol. The server may accept connections from multiple devicessimultaneously. The communication of messages larger than what thehardware can support may still be sent using a data-chunking processbuilt into the protocol, allowing a theoretically unlimited messagesize.

The initial connection setup may start with the server (e.g., the smartmirror 100) advertising as a Bluetooth Low Energy peripheral. The servermay advertise a GATT service with a single GATT write characteristic.Clients scanning for peripherals (e.g., a user's smart device) maylocate and connect to the available GATT service if the client is withinrange of the server (2502). For Bluetooth Low Energy in particular, aclient connected to the server may be identified by its media accesscontrol (MAC) address either (1) ephemerally without bonding where theMAC address is intentionally modified for security purposes or (2)concretely with bonding where a returning client resolves to the sameunique MAC address. The client may be responsible for ensuring that thefirst data written into the characteristic after connection is a 128-bitunique user identification (UUID) in common readable form, base-10representation separated into sections by dashes with 8, 4, 4, 4, and 12digits respectively (2504). Upon a successful write of the UUID, theserver may advertise another GATT service with the written UUID as itsidentifier and a single GATT read characteristic set up to allownotifications. The client may rescan the server's GATT services andcomplete the connection by subscribing to notifications on the readcharacteristic (2506).

Once a connection is established between the client and the server, datamay be sent in both directions (i.e., from the client to the server,from the server to client) using a simple chunking scheme (2508). Thedata exchange may include protocol agnostic raw data thus allowing anapplication developer to set up their own messaging over the availableconnection. To accomplish chunking, each full message is split intoparts that fit within the Bluetooth Low Energy Minimum Transfer Unit(MTU) size, which is determined by the Bluetooth Low Energy connectionsetup and the platform being used.

The chunking process works as follows: (1) the length of the message tobe sent is computed in units of bytes, (2) a preamble of a fixed bytelength that denotes the size of the total message is prepended to theoriginal message, (3) the message with the preamble prepended is splitinto chunks such that they fit within the MTU, (4 a) in the case ofclient to server messaging, the chunks may be written sequentially intothe servers write characteristic, with confirmation as per the BluetoothLow Energy specification, (4 b) in the case of server to clientmessaging, the chunks may be written sequentially into the servers readcharacteristic for the intended recipient, triggering a notification onthe client as per the Bluetooth Low Energy specification, (5) therecipient (i.e., the server or the client) may read the preamble fromthe first chunk to determine the length of the following message, (6)the recipient may fill a buffer with data from successive chunks untilthe expected number of bytes is received, and (7) the message may thenbe parsed/decoded as needed followed by the recipient continuing tolisten for the next preamble.

Connectivity using a HostAP Mode

In some configurations, the smart mirror 100 may be configured to use aHostAP mode (also referred to in the art as the “Chromecast mode” due touse in the Google Chromecast device), which is a method of setting up anInternet of things (IoT) device where the IoT device acts as an accesspoint for other devices using the same interface as a standard wirelessrouter. The HostAP mode may provide several benefits: (1) uses wellknown and field tested/proven methodologies, (2) allows application codeto support one path for messaging, (3) prevents the use of otherinterfaces (i.e., there is no need to use Bluetooth or near-fieldcommunication (NFC)), and (4) allows the application layer to usemulticast Domain Name Systems (mDNS) to discover devices on the networkinstead of other approaches such as Bluetooth scanning, which istypically slow and error prone.

Depending on the operating system of the smart device, a user maymanually change the device's settings to connect to the IoT network. Forexample, a user using a smart device with an iOS operating system, suchas an iPhone or an iPad, should go to the settings of their device toconnect to the IoT devices network for initial setup and/or errorrecovery when the network goes out.

The smart mirror 100 may also be configured to operate the HostAP/mDNSmode. FIGS. 26A-26C show exemplary flow charts of the smart mirror 100integrating the HostAP/mDNS functionality. Specifically, FIG. 26A showsa process 2600 a describing the various states of operation for thesmart mirror 100 and the applicable actions for each state. State (I)represents the smart mirror 100 broadcasting its HostAP network. State(II) represents the ‘Attempt Connection Flow’ process 2600 b, which isdepicted in FIG. 26B. If the smart mirror 100 loses connection to thelocal network over Ethernet or Wi-Fi, the network issue and/or setupprompt screen should be displayed on the display panel 120 of the smartmirror 100. State (III) represents the smart mirror 100 being connectedto the network (Ethernet or Wi-Fi) with no internet access, which isshown to the user in the form of a message. In this case, the messagemay be a full screen blocker on the smart mirror 100. Additionally, ifan unknown client device (e.g., a new user id since the smart mirror 100was installed and configured) connects to the smart mirror 100, theunknown client device should be pin-paired to the smart mirror 100first.

FIG. 26C shows a process 2600 c for a smart device running the iOSoperating system to connect to the smart mirror 100 and/or to setup anetwork connection. The various states of operation and the applicableactions for each state are shown. In particular, state (IV) representsrecoverable error state. As shown, the error states are unlikely tooccur while the smart device is connected to the smart mirror's HostAP,but a user interface/contingency process may be included in the eventsuch a failure does occur.

Live Streaming Content

As described above, the smart mirror 100 is configured to show videocontent on the display panel 120 from a studio (e.g., a fitness studio,a classroom). The video content may be streamed as live content or ason-demand content. For example, live content may be recorded and storedon a central repository such that users may later request and play thevideo content, hence becoming on-demand content. For example, after thevideo content is recorded, the video files may be uploaded to Amazon'sS3 storage and transcoded into moving pictures experts' group dynamicadaptive streaming over HTTP (MPEG-DASH) files. This enables rebroadcastof high-quality, adaptive streaming video to the smart mirror 100. Thesmart mirror 100 may also be configured to receive and/or have access tomultiple live streams that are broadcast simultaneously from multiplestudios and/or sound stages. A user may thus have access to multiplelive streams and through the user interface, may browse and/or selectthe desired live stream. In some configurations, the smart mirror 100may use the Google Android operating system and may thus have access tothe Android Exoplayer library to connect the smart mirror 100 to HTTPlive stream (HLS) streams for a user to view a live workout. Thelive-streaming content may also be set to be publicly viewable and/oraccessible or private (limited to select individuals).

In some applications, the smart mirror 100 may be connected to an onlinestreaming service that provides users with third-party video contentstreamed from a server (e.g., directly through a network router orindirectly through a user's smart device). Third party content may beprovided to users on a subscription basis. The third party may providecontent to a centralized distribution platform, which communicates withthe smart mirror 100 over a network. One benefit of a centralizeddistribution platform is that the distribution of content to the smartmirror 100 is simpler. Alternatively, the third party may develop aseparate distribution platform, which may use separate softwareapplications on the smart device for users to access content.

The smart mirror 100 may be configured to provide video content inaccordance to industry-accepted standards, particularly when handlingvariations in network bandwidth. For video streaming, the smart mirror100 may adhere to the HLS authoring specification, which specifiesconditions for changing the video stream quality in real-time to adaptto a user's network bandwidth. For on-demand video content and encoreworkouts, the smart mirror 100 may adhere to the MPEG-DASHspecification, which also provides conditions for changing video qualityin real-time to adapt to a user's bandwidth. For non-video relatedfunctionalities, the smart mirror 100 may be configured to operate inaccordance to industry-standard mobile development methods, including,but not limited to HTTP request retry logic and user interface (UI)/userexperience (UX) prompts to the user that handle various networkconnectivity and latency issues.

Data Storage and Privacy

The smart mirror 100 may also store user information locally on thesmart mirror 100 and/or a remote storage device (e.g., a cloud service)depending on the amount of storage space used. For example, userinformation that uses little storage space may be stored locally on thesmart mirror 100, including but not limited to the user's name, age,height, weight, and gender. Additionally, video content (e.g., a fitnessroutine) may also be stored the smart mirror 100 to reduce the impact ofnetwork latency, which may affect the video streaming quality. Thisamount of video content stored may be limited by the storage capacity ofthe smart mirror 100. In some configurations, the video content may onlybe stored temporarily on a daily or weekly basis or depending on thepercentage of the smart mirror's capacity being used. User informationthat uses a substantial amount of storage space may be stored on aremote storage device including, but not limited to biometric data, suchas the user's heart rate and breathing rate and video recordings of theuser captured during a workout. The smart mirror 100 may retrieve thisinformation for subsequent analysis and display.

The transfer of data between the smart mirror 100 and a remote storagedevice may be secured (e.g., encrypted) in various ways to preventunwanted loss or theft of user information. For example, the BluetoothLow Energy protocol includes built-in security features that may be usedby devices leveraging this protocol. However, these security featuresmay only be used when a Bluetooth bonding step is completed beforeestablishing a connection with encryption. In some cases, varioussecurity mechanisms may not be implemented or may malfunction, at whichpoint application level security may be implemented in combination withthe chunking specification of data described above. For example,Advanced Encryption Standard (AES) encryption of the message may beapplied before prepending the preamble of a message. In some respects,the Bluetooth Low Energy protocol performs a similar process viabuilt-in security features at a firmware level and may provide similarprotection against a person from reading the communications between aclient and a server.

When a client disconnects from a server, the GATT service added for theclient to read/notify messages may be removed from the service record onthe server's device. This ensures that no connections are left open andthe system does not accidentally leak information to nefarious snoopers.This termination of the connection may be triggered by either the serveror the client and relies on the Bluetooth Low Energy stack to provide anotification to both sides that the connection has closed. If Bluetoothbonding was used in the initial connection setup to provide firmwarelevel encryption security, the bond information may be stored on eachdevice such that bonding does not need to be repeated followingsubsequent connections between the client and the server.

A User Interface for the Smart Mirror

A user may control the smart mirror 100 using a smart phone or tablet orby interfacing directly with the smart mirror 100 (e.g., a voicecommand, a gesture command, a touch command). A graphical user interface(GUI) may be provided to facilitate user interaction with the smartmirror 100. The GUI may be adapted to conform to different user inputsdependent on the manner in which a user interfaces with the smart mirror100. For example, a GUI on a user's smart phone may allow the user tochange settings of the smart mirror 100, select/browse various fitnessclasses, and/or change settings during a workout.

FIG. 27 shows an exemplary GUI displayed on a user's smartphone. The GUImay support touch commands and may be designed to accommodate the sizeof the display on the user's smart phone. In another example, a GUI on auser's computer may provide a more conventional user interface thatrelies upon inputs from a keyboard and/or a mouse. In yet anotherexample, a GUI on the smart mirror 100 may provide voice or gestureprompts to facilitate user-provided voice commands and gesture commands,respectively. The GUI for the smart mirror 100 may be adapted to supportmultiple types of user inputs (e.g., a controller, a remote, a voicecommand, a user command).

The following description provides several exemplary GUI-relatedfeatures to facilitate user interaction with the smart mirror 100. TheseGUI-related features are categorized according to the followingcategories: settings, browsing and selecting a class, class interface,social networking, and background processes. These categories are usedmerely for illustrative purposes and that certain features may beapplied under several situations that may fall under multiple categoriesand/or use cases. One or more of these features may be adapted and/ormodified to accommodate certain user input types. The GUI may extend tomultiple devices including, but not limited to the smart mirror 100, asmart phone, a tablet, a computer, and a remote control.

Smart Mirror Settings

The GUI may allow the user to modify and choose various settings relatedto the operation of the smart mirror 100. For example, the GUI may beused to initially setup a connection between a user's smart device andthe smart mirror 100 (or the smart mirror 100 and a network). FIG. 28Ashows an exemplary GUI screen used to sync a user's smart phone to thesmart mirror 100 and to connect the smart phone and/or smart mirror 100to a network. As shown in FIG. 28A, the GUI may indicate the status ofthe connection of the smart phone and the smart mirror 100 under asettings screen. FIG. 28B shows the GUI may also show the connectionstatus of the smart mirror 100 and brightness of the smart mirrordisplay 120 while using the GUI to navigate and browse for content.Additionally, the GUI may provide prompts to instruct the user the stepsto connect the user's smart device to the smart mirror 100. Generally,the GUI may enable the user to manage the connectivity between the smartmirror 100, the user's smart device, a network router, and anyperipheral devices (e.g., a biometric sensor or a Bluetooth audiodevice).

The GUI may also enable the user to create a user account when firstusing the smart mirror 100. The user account may be used, in part, tomanage and store user information including, but not limited to theuser's name, age, gender, weight, height, fitness goals, injury history,location, workout history, social network blog, contact list, groupmemberships, ratings/reviews of fitness classes, and leaderboard scores.The user account may also be used to store user preferences and accountsettings. In this manner, the user's information may be stored remotely(e.g., on a server or a cloud service), reducing the risk of accidentaldata loss due to failure of the user's smart device or the smart mirror100. The GUI may be configured to have the user log into their accountbefore using the smart mirror 100. The user information may be storedwithout creation of a user account. For example, the user informationmay be stored locally on the user's smart device or the smart mirror100. Depending on the user's settings, the user information may beshared with other users and/or instructors without the use of a useraccount.

The GUI may further include several settings to customize the smartmirror 100 based on the user's preferences. For example, the brightness,contrast, and color temperature (e.g., a warmer hue, a cooler hue) ofthe display panel 120 of the smart mirror 100 may be manually changed inthe GUI. In some cases, these display parameters may be adjustedautomatically depending on ambient lighting conditions and/or userpreferences. For example, the smart mirror 100 may include an ambientlight sensor that monitors ambient lighting conditions, which may beused to adjust the display parameters according to a particularcriteria. For instance, the smart mirror 100 may adjust the display'sbrightness, contrast, color balance, and/or hue, e.g., for increasingvisibility of the video content in bright ambient light or decreasingblue/green light to reduce eye fatigue and/or disruptions to sleepquality during evening hours.

The GUI may enable the user to change the user interface (UI) layout.For example, the GUI may enable the user to toggle the display ofvarious items during a workout including, but not limited to variousbiometric data (e.g., heart rate, step count, etc.), an exercise timer,a feedback survey for a fitness class or each exercise, and a caloriebar (indicating number of calories burned). Some of these options areshown in the exemplary GUI of FIG. 28C. Additionally, the GUI may enablethe user to change the color or theme of the UI including a differentbackground image, font style, and font size. The layout of the GUIduring a workout may also be modified. For example, the size of thevideo content (e.g., the size of the instructor shown on the displaypanel 120) may be changed based on user preferences. In some cases, thesize of the instructor may also be dynamically varied, in part, toaccommodate exercises captured at different viewing angles and/ordifferent levels of magnification.

The GUI may also include options for the user to change their privacysettings. For example, the user may select the type of informationand/or content that is shared with other users. The privacy settings mayallow users to set the level of privacy (e.g., the public, the group,the subgroup, designated contacts, or the user themselves may haveaccess) for different information and/or content. The privacy settingsmay also include what type of information may be stored remotely (e.g.,on a server, a cloud service) or locally on the user's smart device orthe smart mirror 100.

The GUI may also allow the user to adjust various audio settings on thesmart mirror 100 (and/or a speaker peripheral connected to the smartmirror 100/the user's smart device). The audio settings may include, butis not limited to the volume of music, the volume of an instructor'svoice, the volume of another user's voice, and the volume of soundeffects. Additionally, the GUI may allow the user to select languageoptions (e.g., text and audio) and to display subtitles or captionsduring a workout. The GUI may also allow the user to configure aprerecorded voice, which may be used to provide narration, instruction,or prompts. The gender, tone, and style of the prerecorded voice may beadjusted by the user via the GUI.

FIGS. 28D-28F show how the GUI can be used to select and play music withthe smart mirror 100, such as while exercising during a fitness class orwhile the display is off. FIGS. 28D and 28E show the GUI used to connectto and select a music source. The smart mirror 100 may also supportmusic downloaded locally (e.g., onto onboard storage in the smart mirror100) and/or streamed from external sources and third party services,such as Spotify. The music may also be stored on a remote device (e.g.,a smart phone) and transferred to the smart mirror or speaker via awireless or wired connection. The music may be selected independentlyfrom the activity and may be played by the smart mirror 100 or a speakerconnected to the smart mirror 100 (e.g., Bluetooth speaker).Additionally, the music may be arranged and organized as playlists. Theplaylist may be defined by the user, another user, or an instructor.FIG. 28F shows the GUI may support multiple playlists for the user toselect during a given session with the smart mirror 100.

Browsing and Selecting Smart Mirror Classes (Video Content)

The GUI may also enable the user to navigate and browse various contentavailable to be downloaded and/or streamed to the smart mirror 100. TheGUI may generally provide a list of available fitness classes (includingindividual exercises) a user can select. Various types of content may beincluded, such as live streams, recorded video content, and/orcustomized fitness classes. The content may be arranged such thatpertinent information for each class is displayed to the user including,but not limited to the class name, instructor name, duration, skilllevel, date and time (especially if a live stream), user ratings, and apicture of the instructor and/or a representative image of the workout.Once a particular fitness class is selected, additional information onthe class may be displayed to the user including, but not limited to theclass timeline, the class schedule (e.g., types of exercises), names ofother users registered for the class, biometric data of users whopreviously completed the class, a leaderboard, and user reviews. In somecases, a preview video of the class may be shown to the user eitherwithin the list of fitness classes and/or once a particular fitnessclass is selected.

If the content selected by the user is on-demand, the content may beimmediately played on the smart mirror 100 or saved for laterconsumption. If the content is instead a live stream, an integratedcalendar in the GUI may create an entry indicating the date and time thelive fitness class occurs. The calendar may also be configured toinclude entries for on-demand content should the user wish to play thecontent at a later date. The GUI may show the calendar to provide asummary of reserved fitness classes booked by the user. The calendar mayalso be used to determine whether a schedule conflict would occur if theuser selects a class due to an overlap with another class. The GUI mayalso be linked to a user's third party calendar (e.g., a MicrosoftOutlook calendar, a Google calendar, etc.) to provide integration andease of scheduling particularly with other appointments in the user'scalendar.

The GUI may initially list the fitness classes together as a singlelist. The GUI may provide several categories for the user to select inorder to narrow the listing of classes. The GUI may also include one ormore filters to help a user narrow down a selected listing of fitnessclasses to better match the user's preferences. The filter may be basedon various attributes of the user and/or the fitness class including,but not limited to the exercise type, duration, skill level, instructorname, number of registered users, number of openings available, anaverage user score based on registered users and previous users whocompleted the class, injury, location, age, weight, demographic, height,gender, user rating, popularity, date and time, and schedulingavailability.

The GUI may also be configured to provide a listing of the fitnessclasses the user previously attended. This listing may be furthersubdivided between fully completed fitness classes and partiallycompleted fitness classes in case the user wishes to repeat or finish afitness class. The GUI may also provide a listing of the fitness classesthat the user has designated as favorites. Generally, a fitness classmay be favorited before, during, or after the class by selecting aninteractive element configured to designate the content as the user'sfavorite. The GUI may also provide a listing of featured fitness classesto the user. A fitness class may be featured under various conditionsincluding, but not limited to being selected by a moderator or editor,the popularity (e.g., the number of hits for a certain period of time),and the user rating.

Fitness classes may also be recommended to the user. A listing ofrecommended fitness classes may be generated using a combination of theuser's profile and their social network. For example, recommendationsmay be based on various attributes including, but not limited to theuser's age, weight, height, gender, workout history, ratings, favoritedclasses, group membership, contact lists, skill level, workoutperformance, recommendations from other users and/or instructors, andother users that are being followed via the social network component.The recommendations may be updated and further refined based on feedbackprovided by the user. For example, an initial listing of recommendedfitness classes may be shown to the user. The user may then select asubset of the classes that match the user's interest (or don't match theuser's interest). Based on the selection, an updated listing ofrecommended fitness classes may be presented to the user that moreclosely match the selected classes.

FIGS. 29A-29C show an exemplary GUI for the user to browse and select afitness class. FIG. 29A shows a representative listing of fitnessclasses on the user's smart phone. As shown, the class listing mayinclude the time, instructor name, exercise type, and duration. FIG. 29Bshows an exemplary GUI for selecting one or more filters. As shown, thefilters may include workout skill level, duration, instructor, andexercise type. Once a particular class is selected, the GUI may presentadditional information for the class as depicted in FIG. 29C. Forexample, a brief description of the fitness class may be provided.Additionally, biometric data of the user and/or other previous usersattending the class may be displayed to the user to provide anindication of the workout intensity. The GUI may also includeinteractive elements to start and/or resume the fitness class (e.g., inthe event the user previously started the class, but did not finish).

The GUI may also provide the ability to generate customized fitnessclasses designed to better match user preferences. A customized fitnessclass may be constructed from individual exercises extracted frommultiple fitness classes. The type of exercises included may depend onvarious user information including, but not limited to the user'sfitness goals, age, weight, skill level, biometric data, pastperformance, and the types of exercise chosen by the user (e.g., cardio,strength, stretching exercises). Each exercise may also be modifiedaccording to various aspects including, but not limited to the duration,the number of repetitions, and the exercise conditions (e.g.,resistance, weight, incline angle). Additionally, the order of theexercises may be arranged based on the desired pace of the workout. Forexample, a higher intensity workout may place more difficult exercisestogether within the workout. A lower intensity workout may include morerest breaks distributed throughout the workout. The total duration ofthe customized workout may also depend on user preferences including,but not limited to a user-defined duration, the number of calories theuser wishes to burn, and biometric data to determine a preferredduration for the user to meet their fitness goal while reducing the riskof injury (e.g., due to overexertion, dehydration, muscle strain).

Class Interface

Once the user selects the fitness class and the class begins, the GUImay be configured to display various information and/or controls to theuser. As described above, the smart mirror 100 is used primarily to showvideo content via the display panel 120 and audio outputs via thespeakers 152 and 154. In some cases, the display panel 120 may also beconfigured to show GUI-related features that are more informationalrather than a control input. The portion of the GUI with control inputsmay instead be shown on the user's smart device. Therefore, the GUI, asdescribed herein, may be split between the smart mirror 100 and anotherdevice. Of course, the smart mirror 100 may be configured to be usedwithout the aid of another device as described above. In such cases, theinformation and control inputs provided by the GUI may be displayedentirely on the display panel 120.

FIGS. 30A-30C show an exemplary GUI on the user's smart phone used, inpart, to control the fitness class and to provide user input. FIG. 30Ashows the GUI on the user's smart phone may give the user the ability toplay, pause, rewind, fast forward, or skip certain portions of theworkout. The GUI may also include controls for the user to adjust thevolume of the output sound (e.g., from the smart mirror 100 or aBluetooth speaker) and to rate the exercise and/or fitness class. TheGUI on the user's smart phone may also display the current exercise, theskill level, the instructor name, and the duration of the routine. FIG.30B shows an exemplary GUI of a workout log of the user. This workoutlog may be accessed before, during, or after the workout. As shown, theworkout log may contain various information including the total caloriesburned, the total number of workouts, the total duration the user wasexercising, the user's progress in meeting a fitness goal (e.g., aweekly goal), and the number of workouts completed relative to thenumber of workouts to meet the weekly goal. FIG. 30C shows an exemplaryGUI of a survey for the user to provide feedback on the instructorand/or the fitness class.

As described above, the smart mirror 100 may also show variousGUI-related features during the workout. For example, FIG. 31A shows anoverview of the fitness class prior to the start of the workoutincluding video of the instructor, instructor name, skill level,duration, name of the class, brief summary of the class, and timeline.The timeline may be used to indicate the pace and/or intensity level ofclass. For instance, the timeline in FIG. 31A indicates four periods(each represented by two parallel bars) corresponding to a higherintensity workout. In some cases, the timeline may be displayedthroughout the workout on the smart mirror 100 and/or the user's smartdevice. The timeline may also be interactive (on either the smart mirror100 via a touch command or the user's smart device) to allow the user toselect and jump to different sections of the class.

Once the class begins, various GUI-related features may be shown toindicate the status and progress of the user's workout in conjunctionwith the video content. FIG. 31B shows one exemplary GUI on the smartmirror 100 during a workout. As shown, the GUI may include a timerindicating the amount of time passed and a progress bar (e.g.,represented as a circle around the timer) to show the user's progressfor a particular exercise. Depending on the exercise, a counter mayinstead be shown to represent the number of repetitions for theexercise. The GUI also shows the name of the exercise and the number ofusers actively participating in the same fitness class. The GUI may alsoshow the next exercise in the workout. If the user is wearing abiometric sensor, such as a heart rate (HR) monitor, the GUI on thesmart mirror 100 may also display real-time biometric data, such as theuser's heart rate. Additional information derived from the biometricdata may also be displayed, such as the number calories burned based onthe user's heart rate. In some cases, the video content may be augmentedby additional notes from the instructor. For example, FIG. 31B shows theinstructor performing the exercise and a miniaturized representation ofthe instructor performing the same exercise using an alternative formand/or movement. The alternative form may present a more challengingversion of the exercise to the user.

In some cases, the smart mirror 100 may actively monitor the user'sbiometric data to provide additional guidance to the user. For example,FIG. 31C shows the smart mirror 100 may display a message indicating theuser's heart rate has dropped below a desired threshold. Thus, the smartmirror 100 may indicate to the user to increase their intensity in orderto increase their heart rate. In another example. FIG. 31D shows thesmart mirror 100 may inform the user the exercise is modified toaccommodate a user's injury and/or to reduce the risk of injury. Inother cases, the GUI may provide a message containing other informationderived from the biometric data including, but not limited to the user'sheart rate relative to a target heart rate zone, the number of stepsrelative to a target number of steps, the user's perspiration rate, theuser's breathing rate, and the extent to which the user is able toproperly emulate the form and movement of a particular exercise (e.g.,qualified using feedback such as ‘poor’, ‘good’, ‘great’).

The smart mirror 100 may also show avatars corresponding to at least aportion of the other users attending the same fitness class. The avatarmay be an image of each user, an icon, or a graphic. For example, thesmart mirror 100 may acquire an image of the user to display as anavatar during the initial creation of the user's account. The image maybe modified or replaced thereafter. FIGS. 31E-31K show several exemplaryrepresentations of other users' avatars, names, and locations.Additional information from other users may also be shown including, butnot limited to the other users' scores during the workout, skilllevel(s), and biometric data (e.g., heart rate, heart rate relative to atarget heart rate zone, step count).

Other information displayed on the smart mirror 100 may include theuser's heart rate relative to a target heart rate zone. FIGS. 31F-31Kshow a horizontal heart rate range bar 3100 representing a heart raterange. The user's heart rate is shown on the bar in combination with atarget heart rate zone 3102 on the heart rate range bar 3100. Thisinformation may visually indicate whether the user is exerting theappropriate level of intensity during the workout. This heart rateinformation may also be used to compute a score for the user to indicatetheir performance during the workout. For example, FIGS. 31I-31K show ascore bar 3110 indicating the real-time score 3114 of the user relativeto a target score 3112, such as a predetermined score, another user'sscore, the user's previous score when performing the same exerciseand/or workout. The user's score 3114 may change as the exercise orworkout progresses based on the number of points awarded for satisfyingcertain criteria, as discussed below. In some cases, a leaderboard maybe displayed during or after the workout. The leaderboard may rank theusers based on their respective scores.

Once the workout is complete, the GUI may display a summary of theworkout and the weekly exercise log described above. For example, FIG.31L shows the workout log on the smart mirror 100 as previouslydescribed with reference to the GUI shown on the user's smart phone inFIG. 30B. FIG. 31M shows a summary of the workout. As shown, the GUI mayprovide the user's score, the duration the user's heart rate was withinthe target heart rate zone, the user's average heart rate, the number ofcalories burned, and a chart showing the change in the user's heart rateduring the workout. The GUI in FIG. 31M may also show the days of theweek the user met their daily exercise goals.

In some cases, the user may receive achievements during or after theworkout. These achievements may be awarded when the user satisfiescertain criteria, as described below. The achievements may also beshared with other users in the fitness class immediately after receiptor after the workout is complete. Similarly, the user may see anotheruser's achievements during or after the workout. The display ofachievements may be toggled on or off in the settings depending on userpreferences.

As described above, the smart mirror 100 may display a miniaturizedrepresentation of the instructor. In some cases, the miniaturizedrepresentation of the instructor may be overlaid with a correspondingrepresentation of the user captured using the camera 130. Eachrepresentation may be semi-transparent to enable the user to comparetheir form and movement to the instructor. In some cases, therepresentation of the instructor or the representation of the user maybe displayed as a stick model to provide greater visual clarity whencomparing the two representations with respect to one another. In someapplications, the GUI may enable the user to download representations ofother users and/or instructors for guidance when performing a particularexercise. Furthermore, the smart mirror 100 and the GUI may enable theuser to display multiple representations for comparison. For example,representations of each user in a fitness class may be displayed on thesmart mirror 100.

The various GUI-related features shown on the smart mirror 100 may betoggled on or off via the settings GUI described above. The layout,color, and size of these GUI-features may also be customizable. Forexample, the user may wish to show as little information as possible(e.g., only the timer, exercise type, and the progress bar) such thatthe video content and the user's reflection appear less cluttered and/orless obstructed during the workout.

The smart mirror 100 may also be configured to dynamically adjust andadapt content in real-time during a workout. Such adjustments may dependon a combination of user preferences and instructor recommendations. Forexample, the user may specify preferences on various types of fitnessroutines (e.g., cardio workouts, strength training, stretching, upperbody workouts, core workouts, lower body workouts, current injuries, andpast injuries). Based on these preferences, the instructor may recommenda particular set of fitness routines and past user ratings of thesefitness routines.

The recommended fitness routines may be then be streamed to the user andupdated in real-time based on user feedback (e.g., preferences onintensity level of exercise, preferences on exercising certain areas ofthe body). Biometric data (e.g., heart rate, breathing rate) may also bemonitored to adjust the intensity of the fitness routines. For example,the instructor (or the user) may specify a target range for the user'sheart rate during the workout. If the user's heart rate is out of thetarget range, the smart mirror 100 may first warn the user and thenadjust the content to either bring the user's heart rate into the targetrange or modify the target range if the fitness routine is no longerpreferred. Dynamic adaptation of content may be achieved by analyzinguser feedback or biometrics data using a processor with a decision tree,neural network, or another machine learning method.

Sharing Social Media Using a Smart Mirror

The smart mirror 100 may also have a social networking component thatallows the user to connect to another person (e.g., another user, aninstructor) and a group/community of people. The user may connect toanother person using a search feature integrated into the GUI. Thesearch feature may enable the user to search for another person based onvarious attributes including, but not limited to their legal name,username, age, demographic, location, fitness interests, fitness goals,skill level, weight, height, gender, current injuries, injury history,and type of workout music. In one example, once the user selects anotherperson with which they want to connect to, a request may be send to theother person for subsequent confirmation/approval. If the other personapproves, the user may be connected to the other person and may see theperson on a list of contacts. In some cases, the user may configuretheir account to automatically accept requests from other users. Thismay be an option selected under the settings portion of the GUI.

The GUI may also provide other methods for the user to connect toanother person. For example, the user may connect to other users basedon their attendance of a particular fitness class. For example, the usermay register for a fitness class. Before the class begins, the user maybe able to view other users attending the same class. The GUI may enablethe user to select another user and send a connection request. Aconnection request may also be sent during or after the fitness class.The GUI may also recommend people to connect with based on theattributes described above (e.g., the attributes may be combined to forma representation of the user) as well as other attributes including butnot limited to a similar workout history, similar workout performance orprogression, similar scores on a leaderboard, geographic proximity(e.g., based on a user's defined location, an Internet Protocol (IP)address), and/or shared connections with other users (e.g., 1^(st)degree, 2^(nd) degree, 3^(rd) degree connections). The GUI may alsoenable the user to browse through a leaderboard and select another usershown on the leaderboard. Once the other user is selected, a connectionrequest may again be sent.

The GUI may provide a list of contacts to the user, which may be groupedand/or organized according to the user's preferences. For example, thelist of contacts may be arranged based on the user's immediate family,friends, coworkers, list of instructors, people sharing similarinterests, demographic, and so on. The list of contacts may also includea filter that enables the user to select and display one or more groups.

Additionally, the GUI may enable the user to join another group and/orcommunity of users. For example, a user may create a group for usersinterested in cycling. Another group may be created for users interestedin other interests such as boxing, running, weightlifting, and/or yoga.The group may be set to be a public group where any user may see thegroup via the GUI and may send a connection request to join the group.The group may also be set to be a private group that may not beavailable via the GUI and only allows users to join by an invitation.The group may be created by a user or an instructor. Other users mayjoin the group upon approval by the creator or another user withappropriate administrative rights. In some cases, the group may beconfigured to accept all connection request automatically.

The group may be used, in part, to provide users a forum to communicateand share information with one another. For example, a user may providerecommendations for various fitness classes to other users. In anotherexample, an instructor may send a message on a new or upcoming fitnessclass they are teaching. In another example, a user may send a messageindicating they are about to begin a fitness class. The message mayprovide an interactive element that enables other users to join thefitness class directly, thus skipping the various navigational screenspreviously described to select a fitness class. Additionally, a user maypost a message containing audio and/or video acquired by the smartmirror 100 to share with other users in the group. For example, a usermay post a video showing their progress in losing weight. In anotherexample, the user may show video of the instructor and/or other usersparticipating in the fitness class. A user in the group may alsogenerate a group-specific leaderboard to track and rank various membersof the group.

In some cases, the GUI may also enable at least a portion of the userswithin a group to join a particular fitness class together. For example,the users within a group may form a subgroup where a designated leaderof the subgroup may then select a fitness class, using similar processesdescribed above, thus causing the other members of the subgroup toautomatically join the same fitness class. The GUI may also provide liveaudio and/or video chat between users within the same group and/orsubgroup. For example, when a subgroup of users joins a fitness classtogether, the GUI may allow the users of the subgroup to communicatewith one another during the workout. This may include audio and video(e.g., a frame showing the other user's head, face, or body) streamsfrom other users overlaid onto the exercise displayed on the displaypanel 120. It should be appreciated the subgroup may also be formedbased on the user's selection of one or more contacts on their list ofcontacts (as opposed to being restricted to users within a group).

The GUI may also enable the user to create a social network blog toinclude various user-generated content and content automaticallygenerated by the smart mirror 100. User-generated content may include,but is not limited to ratings or reviews of various fitness classes,audio messages generated by the user, video messages generated by theuser, interactive elements linking to one or more fitness classes.Automatically generated content may include, but is not limited toupdates to the user's score on a leaderboard, achievements by the user(e.g., completing a fitness goal), and attendance to a fitness class.The content shown on the user's social network blog may be designated asbeing public (e.g., any user may view the content) or private (e.g.,only select group of users designated by the user may view the content).

The GUI may also enable the user to “follow” another user. In thisdescription, “follow” is defined as the user being able to view anotheruser's information that is publicly accessible including, but notlimited to the other user's social network blog, workout history, andscore(s) on various leaderboards. The option of following another usermay be presented as another option when the user is assessing whetherthey want to connect to other user. Therefore, the GUI may enable theuser to follow another user using similar methods described above in thecontext of connecting to other users.

As described above, the smart mirror 100 may be used to share varioususer information with other users including, but not limited to theuser's profile, social network blog, achievements, biometrics, activityselection, a video recording, and feedback. For example, user A mayshare their progress on a fitness routine to user B, who can thenprovide feedback (e.g., an emoji, an audio message, a video message,etc.) to user A. In another example, the GUI on the smart mirror 100 oron the user's smart phone may prompt the user to take a selfie image,either with the smart mirror's camera 130 or the smart phone, followingthe completion of a workout as shown in FIG. 32A. The camera 130 and thedisplay panel 120 may then be configured to show a live video of theuser to create a desired pose. An image of the user may then be acquired(e.g., after a preset period of time or based on an input command by theuser) as shown in FIG. 32B. The image of the user may then be sharedwith other users (e.g., in the same fitness class, in the user's list ofcontacts, in the user's group) as shown in FIG. 32C. The user may alsoview other user's images.

In another example, the camera 130 may record a video of user A during aworkout, which may then be shared with user B. As user B performs thesame workout, the video of user A may be overlaid onto the display panel120 with a live video of user B. The respective video recording of userA and the live video of user B may be semi-transparent such that user Bmay compare their form and/or movement to user A during the workout. Insome cases, the smart mirror 100 may enable the user to download videorecordings of other users and/or instructors to display onto theirrespective smart mirror 100 whilst performing the workout. In thismanner, the smart mirror 100 may support a “ghost mode” that allowsusers to compare their performance during a workout to other people. Forexample, the user may download a video recording of multiple expertsperforming the same workout. The user may then display the videorecording of each expert (individually or in combination) to evaluatethe user's progression in the workout.

The smart mirror 100 may also support achievements. Achievements aredefined as rewards given to the user upon satisfying certain criteriafor the achievement. The rewards may include, but is not limited to abadge (e.g., a visual graphic the user can share with others), a numberof points contributing to a user's leaderboard position, and access or adiscount to premium content. Achievements may be given for variousreasons including, but not limited to exercising several days in a row,meeting an exercise goal, completing certain types of workouts and/orexercises, completing a certain number of workouts and/or exercises, andadvancing to more difficult skill levels. A summary of the achievementsmay be shown on the GUI to the user.

Information may be shared between users in several ways. In one example,smart mirrors may share data directly with one another via local, directconnections when the smart mirrors are connected to the same network(e.g., multiple smart mirrors at a gym, hotel, or home). In anotherexample, information may be shared via the application installed on eachuser's smart device through a remote network connection (e.g., awireless network, wireless internet, a telecommunication network).Information may also be stored remotely on a server, which may then bedistributed between users (e.g., with or without authorization of theuser depending on the settings of the smart mirror 100 and/or the user'saccount).

Leaderboards, Heart Rate, and Transitions Between Target Heart RateZones

As described above, the GUI may also include one or more leaderboards torank users according to a user's score. For example, a leaderboard maybe generated for each fitness class to rank the participant'sperformance during and after the class. In another example, one or moreglobal leaderboards may be used to rank many, if not all, users based onthe type of exercise or a combination of different exercises.

The leaderboard may be used, in part, to provide a competitiveenvironment when using the smart mirror 100. Users may use their scoresto evaluate their progress at a workout by comparing their currentscores to their own previous scores recorded by the smart mirror 100.Additionally, one user may compete against one or more other users(e.g., globally, within the same group, within the same subgroup) toattain higher scores in a live setting (e.g., users within the samefitness class) or with respect to previous scores recorded by the otheruser(s). The user may configure the leaderboard to show other usersexhibiting similar attributes including, but not limited to demographic,gender, age, height, weight, injury, location, skill level, and fitnessgoal. These attributes may be dependent on the user (e.g., theleaderboard includes users similar to the user) or may be entirelyindependent (e.g., the leaderboard includes users dependent solely onthe criteria specified by the user).

The user's score on a leaderboard may be calculated in various ways. Inone example, the user's score may be determined based on how quickly theuser's hear rate (HR) moves between different target hear rate zones. Atarget hear rate zone may be defined as some percentage range of auser's peak heart rate. Various heart rate zones may thus be definedincluding, but not limited to a rest zone, a fitness zone, an aerobiczone, an anaerobic zone, a fat burn zone, and a cardio zone. There maybe a different target heart rate zone for each section of an exercisevideo (e.g., a warm-up heart rate zone to start, alternating rest andanaerobic heart rate zones during intervals, and a warm-down heart ratezone to end). Depending on the definition of these zones, some zones mayoverlap in the range of the percentage of the user's peak heart rate.

A HR accuracy percentage may be used to determine the number of pointsgiven to the user during a workout. The HR accuracy percentagerepresents how quickly the user's hear rate (HR) moves between thedifferent target hear rate zones. A higher score may correspond to theHR changing instantaneously. However, this situation may be unrealisticand/or may result in an exceedingly challenging condition imposed on theuser resulting in biased scores. In some cases, the score may instead becomputed by comparing the user's heart rate to a HR curve representingthe transitions between different HR zones. The HR curve may include asmoothing effect between each HR zone transition to provide a morerealistically attainable HR accuracy percentage. The smoothing effectmay depend on various metrics including, but not limited to the user'shistorical HR data, the HR data of a community of user's, the userpreferences, the user demographic, the exercise and/or workoutstructure.

In one use case, a single smart mirror 100 may support multiple usersperforming a workout. During the workout the scores for each user may bedisplayed on the display panel 120. In this manner, the users maycompare their scores against one another during the workout, which mayprovide an incentive for the users to achieve a greater workoutperformance compared to the case where each easer exercises on their ownseparately.

In another example, the user's score may be computed based on otherfactors of a user's workout performance (which may depend on the type ofbiometric data collected) including, but not limited to the user's stepcount, number of repetitions for each exercise, distance traveled (e.g.,if running or walking), calories burned, the period of time the user'sHR is in a particular HR zone, the user's form and/or movement whenperforming a particular exercise routine. In some cases, the user'sscore may be modified based on conditions that render a particularexercise more difficult or easier including, but not limited to theweight being lifted, the incline angle of a treadmill, the resistancesetting of an exercise bike, the use of supporting blocks during yoga.These modifications may be in the form of a multiplier applied to theuser's score to rewards and penalize the user based on the relativedifficulty of the conditions of the exercise.

The user's score may be a combination of one or more of the factorsdescribed above. In some cases, the user's score may further includeweights applied to particular exercises to intentionally bias the user'sscore. For example, more points may be awarded to the user forcardio-related exercises compared to strength-related exercises tocorrespond to the user's fitness goal of increasing stamina. The user'sscore may also be computed where strength-related exercises are givenmore points than cardio-related exercises to provide the user a scorerepresentative of their fitness goal of increasing their strength.Multiple scores may thus be generated based on the user's biometric dataand workout history to convey to the user a quantitative metricrepresenting their progress for various fitness attributes.

Smart Mirror Background Processes

In addition to the GUI providing user's the ability to access andcontrol the operation of the smart mirror 100 and/or the content shownon the display panel 120 of the smart mirror 100, various backgroundprocesses may also provide user's additional information when notactively using the smart mirror 100. A background process may be aprocess that performs certain functions that, depending on the output ofthe function, results in the generation and transmission of a message tothe user with a representation of the output. The background process maybe substantially automated. This allows, for instance, a user to runother applications on their smart device while the background process isrunning. These background processes may run locally on a user's smartdevice (e.g., a smart phone) or remotely on a device (e.g., a server)with communication access to the smart mirror 100 and/or the user'ssmart device. A background process may be controlled, in part, via anapplication installed on the user's smart device, the smart mirror 100,and/or a remote device.

The background process may be configured to send various types ofmessages (also treated as notifications) to the user including, but notlimited to a text message, an email, a voicemail, or a post on a user'ssocial network account. The frequency of the messages may vary dependingon the content of the message. For example, a message containing areminder for the user to exercise may be sent every other hour of eachday. In another example, a message from another user or an instructormay be sent to the user immediately after submission or may be storedand aggregated with other messages to be sent as a digest (e.g., anemail digest containing multiple messages). In yet another example,recommendations for fitness classes may be sent to a user on a weekly ormonthly basis. Generally, the message may be sent to the user at variousfrequencies (e.g., ranging between immediately after the message isgenerated to months or even years) depending, in part, on userpreferences. The background process may also be configured to reducepower consumption, thus prolonging a device that operates using abattery (e.g., a user's smartphone or tablet).

In some cases, the message sent by a background process may include aninteractive element (e.g., a web link, a button) for a user to provideinput. For example, a message containing a recommendation for a classmay include one or more options a user can select (e.g., ‘register forclass’, ‘not interested’). If the user selects the option to registerfor the class, a web page or an application may open to a screen thatallows the user to review the class and finalize registration. Inanother example, a message containing a status update of another user(e.g., a user's friend) may provide options for the user to send anemoji to indicate their response. For instance, if the user's friendsuccessfully meets one of their fitness goals, the user may send asmiley face or a thumbs up. In yet another example, a message mayindicate a user's friend is attending a particular fitness class and mayinclude an option to enable the user to join the fitness class withoutnavigating through other screens of the GUI.

Generally, various content may be included in a message generated by abackground process. For example, a background process may monitor theduration of time since a user previously used the smart mirror 100.After exceeding a predefined threshold (e.g., an hour, a day, a week), amessage may be generated to remind the user to exercise. The message mayalso contain a user's progress towards meeting one or more fitnessgoals. The smart mirror 100 may also send status updates to the userincluding, but not limited to when a new software update is availablefor installation, connectivity issues between a user's smart device andthe smart mirror 100 or the smart mirror 100 and a network, anunauthorized login into a user's account, and when another user is usingthe smart mirror 100 (e.g., a family member).

A background process may also relay messages from an instructor (oranother user) sent directly to the user or posted on the user's socialnetwork blog. The message may include, but is not limited to updates onthe status of a fitness class (e.g., cancellation, change of schedule),feedback from an instructor following a particular fitness class,feedback from a personal trainer providing guidance to the user on aregular basis, recommendations for a fitness class, messages posted to acommunity forum, a digest of messages from other users, and/or requestsfor connection on a user's social network. A background process may alsomonitor updates of other user's (e.g., a friend, a person followed bythe user) and send messages in an automated manner when certain updatesoccur. The message may contain various content including, but notlimited to the other user completing one or more fitness goals, theother user registering and/or participating in a new fitness class, theother user liking or providing a high rating to a particular fitnessclass, a change in the other user's position on a leaderboard, updatedpictures of the other user (e.g., after completing a fitness class), andbirthday wishes.

Generating Content for a Smart Mirror

The smart mirror 100 is configured to provide a flexible platform thatallows video content generated by instructors (or other users) to bereadily disseminated to a user. The various networking capabilities ofthe smart mirror 100 described above may enable video content to be livestreamed directly to a user's smart mirror 100 or stored on acentralized distribution platform (e.g., a remote server, a cloudservice) for subsequent consumption by the user. In some cases, thevideo content may be distributed through use of a software applicationconnected to the smart mirror 100 (e.g., a first party app from themanufacturer of the smart mirror 100 or a third party app from astreaming service compatible with the smart mirror 100).

Video content may be generated in various settings, such as a fitnessstudio or a user's home. In one example, an instructor may generatevideo content of a fitness class using a studio. The studio may use astandard, one-camera setup or a more sophisticated setup (e.g., multiplecameras to acquire video at multiple viewing angles) to acquire video ofa fitness class. A producer may be used to monitor and/or control theaudio-visual equipment used to stream the class. The fitness class maybe defined as a single continuous shot from beginning to end of theinstructor performing the workout.

The studio may be setup to stream one class at a time. The video/audioof the class is captured and transcoded via a hardware encoder (e.g.,Epiphan Pearl). The class may be live streamed by uploading the videocontent to a low-latency cloud server (e.g., a Wowza cloud server) wherethe content is transcoded and broadcast to an HLS stream (private orpublic). The content may also be recorded at a high resolution forsubsequent re-use or playback (e.g., as on-demand content).

The studio may include a room with a trainer wall where video isrecorded. The room may have dimensions of approximately 18 feet wide, 30feet long, and 12.3 feet high. The camera recording the video may bepositioned approximately 15 feet from the trainer wall. The camera maybe configured to have a field of view of the trainer area withdimensions approximately 8 feet wide, 9 feet deep, and 9 feet high,corresponding to a recorded area. Other studio arrangements withdifferent room dimensions and camera placement may be used depending onthe desired viewing and field of view of the instructor.

The recorded area may be configured to be a ‘black box’, where the wallsof the recorded area are covered in a dark colored material (e.g., amatte black paint) and the floor is covered with a high grip, darkcolored material (e.g., a black rubber floor with little texture). Therecorded area may be illuminated by side lighting devices disposed atvarious heights and assembled to form a small semi-circle on either sideof the instructor. Overhead lighting may result in reflections and thescattering of light off the floor. In order to maintain a ‘black void’configuration, the presence of overhead lighting should be reduced. Thelighting systems may also be configured to emit light with variouscolors and lighting effects (e.g., highlights, wash effects).

In other settings, the recorded area may be configured to have surfaceswith various colors, patterns, and/or surface finishes including, butnot limited to one or more green screens for picture in picture videos,an all-white backdrop, a black Plexiglas floor, and a grey concretefloor. The studio may also be configured to have one or more cameras torecord video at various angles. For example, two cameras with a B-rollfor a master video or a picture in picture configuration may be used.Two cameras that record video at various angle changes may also be used.

The camera may be configured to record video at various resolutions forlive streaming and/or recording (e.g., 1080p, 1080i, 2K, ultra-highdefinition (UHD), 4K, 8K). The video recorded by the camera may be invarious formats including, but not limited to H.264 and MPEG formats.The video may be recorded at various framerates (e.g., 24 frames persecond). For example, a single fixed camera (Sony FS5K) may be mountedon its side to record video with a portrait view. A look up table (LUT)may be applied to the video feed before being passed to the encoder inorder to reduce the amount of color correction processing during postprocessing of the recorded video. Other various settings on the cameramay be adjusted including, but not limited to International StandardsOrganization (ISO) settings (e.g., ISO 3200), and white balance (WB)settings (e.g., WB 6300K).

For sound recording, one or more microphones may be used. For example,the studio may include two highly directional Audix Miniature shotgunmics mounted on a 50″ gooseneck boom supported by a simplefloor-standing microphone stand. In this manner, environmental noise,such as traffic and external room noise, may be reduced without havingto attach a microphone directly to the instructor. A plurality ofmicrophones may be used to facilitate sound recording when theinstructor changes position. For example, one microphone may beconfigured for instructors in a standing position and another microphonemay be configured for instructors in a crouching/prone position. Anautomixer may be coupled to the microphones to ensure the sound recordedby multiple microphones is properly balanced. Hands-free microphones,such as a lavalier microphone, may also be used for sound recording andto reduce environmental noise during recording. Audio may be recorded atvarious qualities (e.g., AAC 48 kHz stereo 320 kpbs).

The quality of the live stream and the recorded video may besubstantially similar or intentionally different. For example, thestudio may be configured to stream video at a 1080 by 1920 pixelresolution, 24 frames per second, and 23 megabits per second for thehighest bandwidth configuration. Adaptive streaming may also be appliedwhen streaming video to adapt to variabilities in a user's networkbandwidth. The smart mirror 100 may adjust video quality by detectingthe user's bandwidth in real-time.

The fitness class may be recorded in various configurations. Forexample, the fitness class may include only the instructor or theinstructor with one or more students to depending on which configurationprovides improved user immersion and/or a better approach to teaching anexercise technique. This may depend, in part, on user preferences aswell. The video content may be recorded at variable framerates (e.g., ahigh frame rate per second recording may facilitate slow motionplayback). The video content may also be recorded with up to a 360degree format to allow users to change views of the fitness class duringa workout. This effect may also be achieved through use of multiplecameras as well. Furthermore, video content recorded in the studio maybe annotated with exercise specific notes and/or lines drawn on theinstructor to provide greater clarity to the form and movement of aparticular exercise. In some cases, a talking head may also be includedto provide users with narration during the class.

Instructor Interface for Smart Mirror Classes

An instructor user interface may also be provided to assist theinstructor in managing the fitness class in real-time. The instructoruser interface may be shown on the display of various devices including,but not limited to a computer, a smart phone, a tablet, a smart watch,and a television. Furthermore, the instructor user interface may beaccessed using a dedicated software application installed locally on theinstructor's device and/or via a web application using a web browser. Insome cases, the instructor may also use the smart mirror 100 to recordvideo content using the camera 130 and the microphone 160 and/or tomanage the fitness class using the smart mirror 100 directly or a smartdevice coupled to the smart mirror 100.

The instructor user interface may include various information on thefitness class and the users attending the class including, but notlimited to a class itinerary, a class timeline, user information of eachuser, user scores, a leaderboard of users, and user feedback on eachexercise and/or the overall class. The instructor may use the instructoruser interface to select and modify a class plan or timeline before orduring the fitness class. The timeline of the class may be adjusteddynamically in real-time based on the instructor's progress in executingthe class plan. For example, the instructor may decide to shorten orremove a particular exercise in favor of prolonging another exercise. Inanother example, user feedback during the class may indicate to theinstructor the users are getting tired more quickly than anticipated,thus the instructor may change the class plan in favor of lessphysically intense routines.

The instructor user interface may devote one section of the instructor'sdevice display to show the class plan and the class timeline. Anothersection of the instructor user interface may show an instructordashboard with user information of each user. The user information mayinclude, but is not limited to each user's biometric data, user feedback(e.g., emoji's, ratings for each exercise), age, weight, gender, height,injury history, previous fitness classes attended, desired goals for theworkout (e.g., losing weight, building muscle). The instructor dashboardmay also include a summary of the user's attending the class, which maybe updated in real-time. The summary may also include a representativescore of each user as the fitness class progresses. In this manner, theinstructor may determine users who are exceeding or falling behind thepace of the exercise.

The instructor dashboard may also enable the instructor to provideindividual messages and/or feedback to each user in various formatsincluding, but not limited to emojis (e.g., a thumbs up, a thumbs down),audio directed specifically to a particular user or group of users, andvideo directed specifically to a particular user or group of users. Theinstructor may also be able to provide instructions or displays showinghow to perform modified versions of exercises for users who are injuredor who have other physicals limitations. For instance, the instructormay display a main exercise and a modified exercise (e.g., “Squat Jumps”and “Squats” as in FIG. 31H) for those who choose not to perform themain exercise. These versions can be displayed to all users or to onlyaffected users.

As described above, the instructor user interface may be shown on thedisplay panel 120 of the smart mirror 100. The camera 130 and themicrophone 160 of the smart mirror 100 may be used to enable theinstructor to provide the aforementioned feedback to a user or group ofusers. Additionally, the speakers 152 and 154 may be used to receiveaudio feedback from a user or group of users during the fitness class.For example, the instructor may ask how the users are feeling after eachexercise and the users may respond by verbally telling the instructorthe pace is too fast, too slow, or satisfactory.

FIGS. 33A-33C show an exemplary instructor user interface accessedthrough a web browser via an instructor's device, such as a computer.FIG. 33A shows the instructor user interface may include a classschedule with a summary of each exercise and the projected period oftime of each exercise. The class schedule may be configured to show thecurrent exercise, which may be updated in real-time as the fitness classprogresses. The workout schedule in FIG. 33A may also provide controlsto the instructor to pause/resume timers for each exercise, to skipparticular exercises, or to go back and repeat particular exercises.FIG. 33C shows another exemplary class schedule with a magnified view ofthe aforementioned controls available to the instructor.

The instructor user interface may also include a summary of the usersstreaming the fitness class via their respective smart mirror 100. Thesummary may include each user's name, location, and current status basedon one or more emojis. The instructor user interface may include filtersto organize and display users according to various parameters including,but not limited to skill level, fitness goals (e.g., build muscle,de-stress, improve health, improve flexibility, improve definition, loseweight, tone up), current and/or past injuries (e.g., ankle, back, knee,neck, postnatal, prenatal, shoulder, wrist), the user's birthday, theduration of time since a user last worked out (e.g., past 7 days, past30 days).

An instructor may also select individual users in the class to showadditional information for each user as shown in FIG. 33B. Various userinformation may be displayed including, but not limited to the user'spicture, the user's name, the user's location, current and pastinjuries, fitness goals, skill level, weight, birthday, frequency ofuser workouts, workouts with a trainer (e.g., specific to name or typeof trainer), total number of workouts, and user ratings for the class.

Conclusion

All parameters, dimensions, materials, and configurations describedherein are meant to be exemplary and the actual parameters, dimensions,materials, and/or configurations will depend upon the specificapplication or applications for which the inventive teachings is/areused. It is to be understood that the foregoing embodiments arepresented primarily by way of example and that, within the scope of theappended claims and equivalents thereto, inventive embodiments may bepracticed otherwise than as specifically described and claimed.Inventive embodiments of the present disclosure are directed to eachindividual feature, system, article, material, kit, and/or methoddescribed herein.

In addition, any combination of two or more such features, systems,articles, materials, kits, and/or methods, if such features, systems,articles, materials, kits, and/or methods are not mutually inconsistent,is included within the inventive scope of the present disclosure. Othersubstitutions, modifications, changes, and omissions may be made in thedesign, operating conditions and arrangement of respective elements ofthe exemplary implementations without departing from the scope of thepresent disclosure. The use of a numerical range does not precludeequivalents that fall outside the range that fulfill the same function,in the same way, to produce the same result.

The above-described embodiments can be implemented in multiple ways. Forexample, embodiments may be implemented using hardware, software or acombination thereof. When implemented in software, the software code canbe executed on a suitable processor or collection of processors, whetherprovided in a single computer or distributed among multiple computers.

Further, a computer may be embodied in any of a number of forms, such asa rack-mounted computer, a desktop computer, a laptop computer, or atablet computer. Additionally, a computer may be embedded in a devicenot generally regarded as a computer but with suitable processingcapabilities, including a Personal Digital Assistant (PDA), a smartphone or any other suitable portable or fixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audible format.

Such computers may be interconnected by one or more networks in asuitable form, including a local area network or a wide area network,such as an enterprise network, an intelligent network (IN) or theInternet. Such networks may be based on a suitable technology, mayoperate according to a suitable protocol, and may include wirelessnetworks, wired networks or fiber optic networks.

The various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Additionally, suchsoftware may be written using any of a number of suitable programminglanguages and/or programming or scripting tools, and also may becompiled as executable machine language code or intermediate code thatis executed on a framework or virtual machine. Some implementations mayspecifically employ one or more of a particular operating system orplatform and a particular programming language and/or scripting tool tofacilitate execution.

Also, various inventive concepts may be embodied as one or more methods,of which at least one example has been provided. The acts performed aspart of the method may in some instances be ordered in different ways.Accordingly, in some inventive implementations, respective acts of agiven method may be performed in an order different than specificallyillustrated, which may include performing some acts simultaneously (evenif such acts are shown as sequential acts in illustrative embodiments).

All publications, patent applications, patents, and other referencesmentioned herein are incorporated by reference in their entirety.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of” “only one of” or“exactly one of.” “Consisting essentially of” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

In the claims, as well as in the specification above, all transitionalphrases such as “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” “holding,” “composed of,” and the like are tobe understood to be open-ended, i.e., to mean including but not limitedto. Only the transitional phrases “consisting of” and “consistingessentially of” shall be closed or semi-closed transitional phrases,respectively, as set forth in the United States Patent Office Manual ofPatent Examining Procedures, Section 2111.03.

1. An apparatus, comprising: a processor; a camera operably coupled tothe processor; and a memory operably coupled to the processor andstoring processor-readable instructions to cause the processor to:receive, from the camera, real-time video data for a user performing anexercise associated with a first exercise class; cause display of thereal-time video of the user via a display in communication with theapparatus while video of an instructor is displayed via the display, toprovide a visual comparison of the user to the instructor; perform imageanalysis of the real-time video of the user to determine a performanceof the user when the user is performing the exercise; cause display of arepresentation of the performance of the user via the display; receive,from a wearable device including a biometric sensor, first biometricdata associated with the user; identify, at a first time, and based onthe first biometric data associated with the user, a first heart rate ofthe user; identify, after the first time, a first score for the userbased on the first heart rate of the user; cause display of the firstscore via the display; receive, from the wearable device, secondbiometric data associated with the user; identify, at a second time, andbased on the second biometric data associated with the user, a secondheart rate of the user; identify, after the second time, a second scorefor the user based on the second heart rate of the user; cause displayof the second score via the display; determine a recommendation for asecond exercise class different from the first exercise class based on aprofile of the user; and cause display of a representation of therecommendation via the display.
 2. The apparatus of claim 1, wherein thecamera is a spatial motion sensing camera.
 3. The apparatus of claim 1,wherein the readable instructions to cause the processor to performimage analysis include instructions to cause the processor to track anindividual extremity of the user.
 4. The apparatus of claim 1, whereinthe memory further stores processor-readable instructions to cause theprocessor to cause display, via the display, of a number of caloriesburned.
 5. The apparatus of claim 1, wherein the wearable device ispositioned on an extremity of the user when the first biometric dataassociated with the user is received.
 6. The apparatus of claim 1,wherein the memory further stores processor-readable instructions tocause the processor to: identify a heart rate zone associated with theuser; and cause display of the heart rate zone via the display.
 7. Theapparatus of claim 1, wherein at least one of the identifying the firstscore or the identifying the second score is performed using machinelearning.
 8. The apparatus of claim 1, wherein the memory further storesprocessor-readable instructions to cause the processor to cause display,via the display, of a horizontal heart rate range bar.
 9. The apparatusof claim 1, wherein the memory further stores processor-readableinstructions to cause the processor to cause display, via the display,of a representation of a percentage of a peak heart rate of the user.10. The apparatus of claim 1, wherein the memory further storesprocessor-readable instructions to cause the processor to cause display,via the display, of a horizontal heart rate range bar and arepresentation of a percentage of a peak heart rate of the user.
 11. Anapparatus, comprising: a processor; a camera operably coupled to theprocessor; and a memory operably coupled to the processor and storingprocessor-readable instructions to cause the processor to: receive, fromthe camera, real-time video data for a user performing an exerciseassociated with a first exercise class; cause display of the real-timevideo of the user via a display in communication with the processorwhile video of an instructor is displayed via the display, to provide avisual comparison of the user to the instructor; receive, from awearable device including a biometric sensor, biometric data associatedwith the user; analyze, using machine learning, the biometric dataassociated with the user to determine a first performance of the user;analyze, at a second time, the real-time video of the user to determinea second performance of the user; identify a score for the user based onthe second performance of the user; cause display of at least one of thescore, an indication of the first performance, or an indication of thesecond performance via the display; determine a recommendation for asecond exercise class different from the first exercise class; and causedisplay of a representation of the recommendation via the display. 12.The apparatus of claim 11, wherein the camera is a spatial motionsensing camera.
 13. The apparatus of claim 11, wherein the camera isconfigured to track an individual extremity of the user.
 14. Theapparatus of claim 11, wherein the memory further storesprocessor-readable instructions to cause the processor to cause display,via the display, of a number of calories burned.
 15. The apparatus ofclaim 11, wherein the wearable device is positioned on an extremity ofthe user when the biometric data associated with the user is received.16. The apparatus of claim 11, wherein the memory further storesprocessor-readable instructions to cause the processor to : identify aheart rate zone associated with the user based on the biometric data;and cause display of the heart rate zone via the display.
 17. Theapparatus of claim 11, wherein the instructions to cause the processorto identify the score include instructions to cause the processor toidentify the score using machine learning.
 18. The apparatus of claim11, wherein at least one of the indication of the first performance orthe indication of the second performance includes a horizontal heartrate range bar.
 19. The apparatus of claim 11, wherein at least one ofthe indication of the first performance or the indication of the secondperformance includes a representation of a percentage of a peak heartrate of the user.
 20. The apparatus of claim 11, wherein at least one ofthe indication of the first performance or the indication of the secondperformance includes a horizontal heart rate range bar and arepresentation of a percentage of a peak heart rate of the user.
 21. Theapparatus of claim 11, wherein the display is controllable via voicecommands.
 22. An apparatus, comprising: a processor; and a memoryoperably coupled to the processor and storing processor-readableinstructions to cause the processor to: receive real-time video data fora user performing an exercise associated with a first exercise class;cause display of the real-time video of the user via a display incommunication with the IoT device while video of an instructor isdisplayed via the display, to provide a visual comparison of the user tothe instructor; perform image analysis of the real-time video of theuser to determine a performance of the user when the user is performingthe exercise; modify the display of the real-time video of the userbased on the performance of the user; receive, from a wearable deviceincluding a biometric sensor, first biometric data associated with theuser; identify, at a first time, and based on the first biometric dataassociated with the user, a first heart rate of the user; identify,after the first time, a first score for the user based on the firstheart rate of the user; cause display of the first score via thedisplay; receive, from the wearable device, second biometric dataassociated with the user; identify, at a second time, and based on thesecond biometric data associated with the user, a second heart rate ofthe user; identify, after the second time, a second score for the userbased on the second heart rate of the user; cause display of the secondscore via the display; determine a recommendation for a second exerciseclass different from the first exercise class based on a profile of theuser; and cause display of a representation of the recommendation viathe display.
 23. The apparatus of claim 22, further comprising a spatialmotion sensing camera.
 24. The apparatus of claim 22, whereininstructions to cause the processor to identify the first score includeinstructions to cause the processor to identify the first score usingmachine learning.
 25. The apparatus of claim 22, wherein the memoryfurther stores processor-readable instructions to cause the processor tocause display, via the display, of a horizontal heart range bar.
 26. Theapparatus of claim 22, wherein the instructions to cause the processorto perform image analysis include instructions to cause the processor totrack an individual extremity of the user.
 27. An apparatus, comprising:a processor; and a memory operably coupled to the processor and storingprocessor-readable instructions to cause the processor to: receivereal-time video data for a user performing an exercise associated with afirst exercise class; cause display of the real-time video of the uservia a display in communication with the IoT device while video of aninstructor is displayed via the display, to provide a visual comparisonof the user to the instructor; perform image analysis of the real-timevideo of the user to determine a performance of the user when the useris performing the exercise; cause display of an adjusted intensity ofthe exercise based on the performance of the user; receive, from awearable device including a biometric sensor, first biometric dataassociated with the user; identify, at a first time, and based on thefirst biometric data associated with the user, a first heart rate of theuser; identify, after the first time, a first score for the user basedon the first heart rate of the user; cause display of the first scorevia the display; receive, from the wearable device, second biometricdata associated with the user; identify, at a second time and based onthe second biometric data associated with the user, a second heart rateof the user; identify, after the second time, a second score for theuser based on the second heart rate of the user; cause display of thesecond score via the display; determine a recommendation for a secondexercise class different from the first exercise class based on aprofile of the user; and cause display of a representation of therecommendation via the display.
 28. The apparatus of claim 27, whereinthe display is controllable via voice commands.
 29. The apparatus ofclaim 27, wherein the memory further stores processor-readableinstructions to cause the processor to cause display of a representationof at least one of the first heart rate of the user or the second heartrate of the user via the display.
 30. The apparatus of claim 29, whereinthe representation of the at least one of the first heart rate of theuser or the second heart rate of the user includes a horizontal heartrange bar.